2023-03-23 19:53:32,078 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101 2023-03-23 19:53:32,092 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-23 19:53:32,112 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-23 19:53:32,113 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6, deleteOnExit=true 2023-03-23 19:53:32,113 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-23 19:53:32,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/test.cache.data in system properties and HBase conf 2023-03-23 19:53:32,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/hadoop.tmp.dir in system properties and HBase conf 2023-03-23 19:53:32,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/hadoop.log.dir in system properties and HBase conf 2023-03-23 19:53:32,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-23 19:53:32,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-23 19:53:32,116 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-23 19:53:32,234 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-23 19:53:32,666 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-23 19:53:32,672 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-23 19:53:32,672 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-23 19:53:32,673 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-23 19:53:32,673 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-23 19:53:32,674 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-23 19:53:32,674 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-23 19:53:32,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-23 19:53:32,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-23 19:53:32,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-23 19:53:32,676 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/nfs.dump.dir in system properties and HBase conf 2023-03-23 19:53:32,677 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/java.io.tmpdir in system properties and HBase conf 2023-03-23 19:53:32,677 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-23 19:53:32,678 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-23 19:53:32,678 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-23 19:53:33,297 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-23 19:53:33,301 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-23 19:53:33,598 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-23 19:53:33,784 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-23 19:53:33,805 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-23 19:53:33,846 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-23 19:53:33,878 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/java.io.tmpdir/Jetty_localhost_46737_hdfs____.ei5irh/webapp 2023-03-23 19:53:34,030 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46737 2023-03-23 19:53:34,050 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-23 19:53:34,051 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-23 19:53:34,613 WARN [Listener at localhost/46107] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-23 19:53:34,688 WARN [Listener at localhost/46107] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-23 19:53:34,713 WARN [Listener at localhost/46107] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-23 19:53:34,721 INFO [Listener at localhost/46107] log.Slf4jLog(67): jetty-6.1.26 2023-03-23 19:53:34,728 INFO [Listener at localhost/46107] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/java.io.tmpdir/Jetty_localhost_42559_datanode____zfr2ki/webapp 2023-03-23 19:53:34,860 INFO [Listener at localhost/46107] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42559 2023-03-23 19:53:35,264 WARN [Listener at localhost/45187] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-23 19:53:35,782 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x57b8dde54d7efefb: Processing first storage report for DS-40542ef7-b74c-48da-99cc-934dd6fbe16a from datanode 8951c20b-99ff-4f66-9662-9fe0fcff31a4 2023-03-23 19:53:35,784 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x57b8dde54d7efefb: from storage DS-40542ef7-b74c-48da-99cc-934dd6fbe16a node DatanodeRegistration(127.0.0.1:44787, datanodeUuid=8951c20b-99ff-4f66-9662-9fe0fcff31a4, infoPort=37173, infoSecurePort=0, ipcPort=45187, storageInfo=lv=-57;cid=testClusterID;nsid=612796149;c=1679601213371), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-23 19:53:35,784 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x57b8dde54d7efefb: Processing first storage report for DS-97e2a041-d2d1-4dc8-b584-bbf36faa532b from datanode 8951c20b-99ff-4f66-9662-9fe0fcff31a4 2023-03-23 19:53:35,784 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x57b8dde54d7efefb: from storage DS-97e2a041-d2d1-4dc8-b584-bbf36faa532b node DatanodeRegistration(127.0.0.1:44787, datanodeUuid=8951c20b-99ff-4f66-9662-9fe0fcff31a4, infoPort=37173, infoSecurePort=0, ipcPort=45187, storageInfo=lv=-57;cid=testClusterID;nsid=612796149;c=1679601213371), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-23 19:53:35,871 DEBUG [Listener at localhost/45187] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101 2023-03-23 19:53:35,945 INFO [Listener at localhost/45187] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6/zookeeper_0, clientPort=60301, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-23 19:53:35,960 INFO [Listener at localhost/45187] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60301 2023-03-23 19:53:35,967 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:35,970 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:36,632 INFO [Listener at localhost/45187] util.FSUtils(471): Created version file at hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78 with version=8 2023-03-23 19:53:36,633 INFO [Listener at localhost/45187] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/hbase-staging 2023-03-23 19:53:36,959 INFO [Listener at localhost/45187] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-23 19:53:37,451 INFO [Listener at localhost/45187] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-23 19:53:37,483 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:37,484 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:37,484 INFO [Listener at localhost/45187] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-23 19:53:37,484 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:37,484 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-23 19:53:37,629 INFO [Listener at localhost/45187] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-23 19:53:37,705 DEBUG [Listener at localhost/45187] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-23 19:53:37,806 INFO [Listener at localhost/45187] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39351 2023-03-23 19:53:37,817 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:37,819 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:37,841 INFO [Listener at localhost/45187] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39351 connecting to ZooKeeper ensemble=127.0.0.1:60301 2023-03-23 19:53:37,888 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:393510x0, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-23 19:53:37,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39351-0x10112f0937e0000 connected 2023-03-23 19:53:37,925 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-23 19:53:37,926 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-23 19:53:37,929 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-23 19:53:37,937 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39351 2023-03-23 19:53:37,938 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39351 2023-03-23 19:53:37,938 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39351 2023-03-23 19:53:37,939 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39351 2023-03-23 19:53:37,939 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39351 2023-03-23 19:53:37,945 INFO [Listener at localhost/45187] master.HMaster(444): hbase.rootdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78, hbase.cluster.distributed=false 2023-03-23 19:53:38,016 INFO [Listener at localhost/45187] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-23 19:53:38,016 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:38,016 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:38,017 INFO [Listener at localhost/45187] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-23 19:53:38,017 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-23 19:53:38,017 INFO [Listener at localhost/45187] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-23 19:53:38,022 INFO [Listener at localhost/45187] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-23 19:53:38,025 INFO [Listener at localhost/45187] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34277 2023-03-23 19:53:38,028 INFO [Listener at localhost/45187] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-23 19:53:38,033 DEBUG [Listener at localhost/45187] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-23 19:53:38,034 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:38,038 INFO [Listener at localhost/45187] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:38,040 INFO [Listener at localhost/45187] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34277 connecting to ZooKeeper ensemble=127.0.0.1:60301 2023-03-23 19:53:38,043 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:342770x0, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-23 19:53:38,044 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): regionserver:342770x0, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-23 19:53:38,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34277-0x10112f0937e0001 connected 2023-03-23 19:53:38,046 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-23 19:53:38,047 DEBUG [Listener at localhost/45187] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-23 19:53:38,048 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34277 2023-03-23 19:53:38,048 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34277 2023-03-23 19:53:38,049 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34277 2023-03-23 19:53:38,049 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34277 2023-03-23 19:53:38,052 DEBUG [Listener at localhost/45187] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34277 2023-03-23 19:53:38,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-23 19:53:38,067 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-23 19:53:38,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-23 19:53:38,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-23 19:53:38,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39351,1679601216784 from backup master directory 2023-03-23 19:53:38,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-23 19:53:38,092 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,092 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-23 19:53:38,093 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-23 19:53:38,093 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,096 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-23 19:53:38,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-23 19:53:38,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/hbase.id with ID: 10fe0f22-adfc-44fc-b286-5a601e2b96fc 2023-03-23 19:53:38,221 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-23 19:53:38,235 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,278 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x237d4a4e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:38,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a3f0b8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:38,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:53:38,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-23 19:53:38,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-23 19:53:38,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-23 19:53:38,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:38,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:38,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-23 19:53:38,423 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store-tmp 2023-03-23 19:53:38,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:38,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-23 19:53:38,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:53:38,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:53:38,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-23 19:53:38,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:53:38,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:53:38,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-23 19:53:38,461 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/WALs/jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39351%2C1679601216784, suffix=, logDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/WALs/jenkins-hbase4.apache.org,39351,1679601216784, archiveDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/oldWALs, maxLogs=10 2023-03-23 19:53:38,552 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK] 2023-03-23 19:53:38,560 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:38,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/WALs/jenkins-hbase4.apache.org,39351,1679601216784/jenkins-hbase4.apache.org%2C39351%2C1679601216784.1679601218501 2023-03-23 19:53:38,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK]] 2023-03-23 19:53:38,622 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:53:38,622 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:38,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,628 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,690 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,701 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-23 19:53:38,726 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-23 19:53:38,740 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:38,747 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-23 19:53:38,772 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:53:38,773 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71837321, jitterRate=0.07045949995517731}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-23 19:53:38,773 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-23 19:53:38,774 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-23 19:53:38,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-23 19:53:38,797 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-23 19:53:38,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-23 19:53:38,802 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-23 19:53:38,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-03-23 19:53:38,836 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-23 19:53:38,861 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-23 19:53:38,867 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-23 19:53:38,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-23 19:53:38,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-23 19:53:38,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-23 19:53:38,911 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-23 19:53:38,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-23 19:53:38,918 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,919 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-23 19:53:38,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-23 19:53:38,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-23 19:53:38,940 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-23 19:53:38,940 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-23 19:53:38,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,942 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39351,1679601216784, sessionid=0x10112f0937e0000, setting cluster-up flag (Was=false) 2023-03-23 19:53:38,962 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-23 19:53:38,976 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:38,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-23 19:53:38,989 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:38,991 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.hbase-snapshot/.tmp 2023-03-23 19:53:39,056 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(952): ClusterId : 10fe0f22-adfc-44fc-b286-5a601e2b96fc 2023-03-23 19:53:39,061 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-23 19:53:39,066 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-23 19:53:39,067 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-23 19:53:39,070 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-23 19:53:39,071 DEBUG [RS:0;jenkins-hbase4:34277] zookeeper.ReadOnlyZKClient(139): Connect 0x1c47425d to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:39,079 DEBUG [RS:0;jenkins-hbase4:34277] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1280bfa5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:39,079 DEBUG [RS:0;jenkins-hbase4:34277] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16bf292d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-23 19:53:39,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-23 19:53:39,111 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34277 2023-03-23 19:53:39,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-23 19:53:39,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,114 INFO [RS:0;jenkins-hbase4:34277] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-23 19:53:39,114 INFO [RS:0;jenkins-hbase4:34277] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-23 19:53:39,115 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1023): About to register with Master. 2023-03-23 19:53:39,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679601249116 2023-03-23 19:53:39,119 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-23 19:53:39,119 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39351,1679601216784 with isa=jenkins-hbase4.apache.org/172.31.14.131:34277, startcode=1679601218015 2023-03-23 19:53:39,122 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-23 19:53:39,123 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-23 19:53:39,128 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-23 19:53:39,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-23 19:53:39,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-23 19:53:39,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-23 19:53:39,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-23 19:53:39,138 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-23 19:53:39,139 DEBUG [RS:0;jenkins-hbase4:34277] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-23 19:53:39,141 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-23 19:53:39,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-23 19:53:39,145 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-23 19:53:39,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-23 19:53:39,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-23 19:53:39,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679601219149,5,FailOnTimeoutGroup] 2023-03-23 19:53:39,151 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679601219150,5,FailOnTimeoutGroup] 2023-03-23 19:53:39,151 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,151 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-23 19:53:39,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,194 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52927, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-23 19:53:39,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,224 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78 2023-03-23 19:53:39,224 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:46107 2023-03-23 19:53:39,224 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-23 19:53:39,229 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-23 19:53:39,230 DEBUG [RS:0;jenkins-hbase4:34277] zookeeper.ZKUtil(162): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,232 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34277,1679601218015] 2023-03-23 19:53:39,232 WARN [RS:0;jenkins-hbase4:34277] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-23 19:53:39,233 INFO [RS:0;jenkins-hbase4:34277] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-23 19:53:39,233 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1947): logDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/WALs/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,241 DEBUG [RS:0;jenkins-hbase4:34277] zookeeper.ZKUtil(162): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,251 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-23 19:53:39,259 INFO [RS:0;jenkins-hbase4:34277] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-23 19:53:39,278 INFO [RS:0;jenkins-hbase4:34277] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-23 19:53:39,283 INFO [RS:0;jenkins-hbase4:34277] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-23 19:53:39,283 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,284 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-23 19:53:39,292 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,293 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,293 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,293 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,293 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,293 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,294 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-23 19:53:39,294 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,294 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,294 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,294 DEBUG [RS:0;jenkins-hbase4:34277] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-23 19:53:39,295 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,295 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,295 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,311 INFO [RS:0;jenkins-hbase4:34277] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-23 19:53:39,313 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34277,1679601218015-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:39,329 INFO [RS:0;jenkins-hbase4:34277] regionserver.Replication(203): jenkins-hbase4.apache.org,34277,1679601218015 started 2023-03-23 19:53:39,329 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34277,1679601218015, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34277, sessionid=0x10112f0937e0001 2023-03-23 19:53:39,329 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-23 19:53:39,329 DEBUG [RS:0;jenkins-hbase4:34277] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,330 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34277,1679601218015' 2023-03-23 19:53:39,330 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:39,330 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34277,1679601218015' 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-23 19:53:39,331 DEBUG [RS:0;jenkins-hbase4:34277] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-23 19:53:39,332 DEBUG [RS:0;jenkins-hbase4:34277] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-23 19:53:39,332 INFO [RS:0;jenkins-hbase4:34277] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-23 19:53:39,332 INFO [RS:0;jenkins-hbase4:34277] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-23 19:53:39,443 INFO [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34277%2C1679601218015, suffix=, logDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/WALs/jenkins-hbase4.apache.org,34277,1679601218015, archiveDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/oldWALs, maxLogs=32 2023-03-23 19:53:39,460 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK] 2023-03-23 19:53:39,464 INFO [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/WALs/jenkins-hbase4.apache.org,34277,1679601218015/jenkins-hbase4.apache.org%2C34277%2C1679601218015.1679601219445 2023-03-23 19:53:39,464 DEBUG [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK]] 2023-03-23 19:53:39,569 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-23 19:53:39,571 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-23 19:53:39,571 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78 2023-03-23 19:53:39,985 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:39,988 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-23 19:53:39,990 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/info 2023-03-23 19:53:39,991 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-23 19:53:39,993 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:39,993 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-23 19:53:39,996 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/rep_barrier 2023-03-23 19:53:39,997 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-23 19:53:39,997 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:39,998 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-23 19:53:40,000 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/table 2023-03-23 19:53:40,001 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-23 19:53:40,001 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:40,003 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740 2023-03-23 19:53:40,004 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740 2023-03-23 19:53:40,007 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:53:40,009 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-23 19:53:40,012 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:53:40,013 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60058889, jitterRate=-0.10505281388759613}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:53:40,014 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-23 19:53:40,014 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-23 19:53:40,014 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-23 19:53:40,014 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-23 19:53:40,014 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-23 19:53:40,014 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-23 19:53:40,015 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-23 19:53:40,015 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-23 19:53:40,020 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-23 19:53:40,020 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-23 19:53:40,028 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-23 19:53:40,043 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-23 19:53:40,046 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-23 19:53:40,198 DEBUG [jenkins-hbase4:39351] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-23 19:53:40,201 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34277,1679601218015, state=OPENING 2023-03-23 19:53:40,209 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-23 19:53:40,210 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:40,211 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-23 19:53:40,214 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:53:40,398 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:40,400 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-23 19:53:40,404 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-23 19:53:40,417 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-23 19:53:40,418 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-23 19:53:40,422 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34277%2C1679601218015.meta, suffix=.meta, logDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/WALs/jenkins-hbase4.apache.org,34277,1679601218015, archiveDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/oldWALs, maxLogs=32 2023-03-23 19:53:40,441 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK] 2023-03-23 19:53:40,447 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/WALs/jenkins-hbase4.apache.org,34277,1679601218015/jenkins-hbase4.apache.org%2C34277%2C1679601218015.meta.1679601220423.meta 2023-03-23 19:53:40,447 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44787,DS-40542ef7-b74c-48da-99cc-934dd6fbe16a,DISK]] 2023-03-23 19:53:40,448 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:53:40,450 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-23 19:53:40,469 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-23 19:53:40,476 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-23 19:53:40,482 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-23 19:53:40,482 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:40,482 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-23 19:53:40,483 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-23 19:53:40,486 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-23 19:53:40,487 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/info 2023-03-23 19:53:40,487 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/info 2023-03-23 19:53:40,488 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-23 19:53:40,489 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:40,489 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-23 19:53:40,491 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/rep_barrier 2023-03-23 19:53:40,491 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/rep_barrier 2023-03-23 19:53:40,492 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-23 19:53:40,493 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:40,493 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-23 19:53:40,494 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/table 2023-03-23 19:53:40,494 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/table 2023-03-23 19:53:40,495 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-23 19:53:40,496 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:40,497 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740 2023-03-23 19:53:40,500 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740 2023-03-23 19:53:40,503 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:53:40,505 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-23 19:53:40,506 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71719653, jitterRate=0.06870611011981964}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:53:40,507 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-23 19:53:40,517 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679601220390 2023-03-23 19:53:40,536 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-23 19:53:40,537 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-23 19:53:40,537 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34277,1679601218015, state=OPEN 2023-03-23 19:53:40,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-23 19:53:40,540 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-23 19:53:40,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-23 19:53:40,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34277,1679601218015 in 326 msec 2023-03-23 19:53:40,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-23 19:53:40,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 517 msec 2023-03-23 19:53:40,555 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5180 sec 2023-03-23 19:53:40,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679601220555, completionTime=-1 2023-03-23 19:53:40,556 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-23 19:53:40,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-23 19:53:40,620 DEBUG [hconnection-0x6185653f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:40,624 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46754, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:40,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-23 19:53:40,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679601280643 2023-03-23 19:53:40,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679601340643 2023-03-23 19:53:40,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-03-23 19:53:40,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1679601216784-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:40,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1679601216784-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:40,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1679601216784-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:40,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39351, period=300000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:40,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-23 19:53:40,678 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-23 19:53:40,686 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-23 19:53:40,687 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-23 19:53:40,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-23 19:53:40,699 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:53:40,701 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:53:40,720 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/hbase/namespace/72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:40,722 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/hbase/namespace/72272839a6ffef279ba4e656fef71504 empty. 2023-03-23 19:53:40,723 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/hbase/namespace/72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:40,723 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-23 19:53:41,147 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-23 19:53:41,149 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 72272839a6ffef279ba4e656fef71504, NAME => 'hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:53:41,562 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:41,563 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 72272839a6ffef279ba4e656fef71504, disabling compactions & flushes 2023-03-23 19:53:41,563 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,563 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,563 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. after waiting 0 ms 2023-03-23 19:53:41,563 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,563 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,563 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 72272839a6ffef279ba4e656fef71504: 2023-03-23 19:53:41,569 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:53:41,587 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679601221572"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601221572"}]},"ts":"1679601221572"} 2023-03-23 19:53:41,615 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:53:41,617 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:53:41,621 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601221617"}]},"ts":"1679601221617"} 2023-03-23 19:53:41,626 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-23 19:53:41,636 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=72272839a6ffef279ba4e656fef71504, ASSIGN}] 2023-03-23 19:53:41,639 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=72272839a6ffef279ba4e656fef71504, ASSIGN 2023-03-23 19:53:41,641 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=72272839a6ffef279ba4e656fef71504, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:53:41,793 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=72272839a6ffef279ba4e656fef71504, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:41,793 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679601221792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601221792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601221792"}]},"ts":"1679601221792"} 2023-03-23 19:53:41,798 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 72272839a6ffef279ba4e656fef71504, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:53:41,959 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,960 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72272839a6ffef279ba4e656fef71504, NAME => 'hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:53:41,960 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:41,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,963 INFO [StoreOpener-72272839a6ffef279ba4e656fef71504-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,965 DEBUG [StoreOpener-72272839a6ffef279ba4e656fef71504-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/info 2023-03-23 19:53:41,965 DEBUG [StoreOpener-72272839a6ffef279ba4e656fef71504-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/info 2023-03-23 19:53:41,966 INFO [StoreOpener-72272839a6ffef279ba4e656fef71504-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72272839a6ffef279ba4e656fef71504 columnFamilyName info 2023-03-23 19:53:41,967 INFO [StoreOpener-72272839a6ffef279ba4e656fef71504-1] regionserver.HStore(310): Store=72272839a6ffef279ba4e656fef71504/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:41,969 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,970 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,975 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 72272839a6ffef279ba4e656fef71504 2023-03-23 19:53:41,979 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:53:41,980 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 72272839a6ffef279ba4e656fef71504; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71949026, jitterRate=0.07212403416633606}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-23 19:53:41,980 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 72272839a6ffef279ba4e656fef71504: 2023-03-23 19:53:41,982 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504., pid=6, masterSystemTime=1679601221952 2023-03-23 19:53:41,986 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,986 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:53:41,987 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=72272839a6ffef279ba4e656fef71504, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:41,987 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679601221986"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601221986"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601221986"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601221986"}]},"ts":"1679601221986"} 2023-03-23 19:53:41,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-23 19:53:41,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 72272839a6ffef279ba4e656fef71504, server=jenkins-hbase4.apache.org,34277,1679601218015 in 193 msec 2023-03-23 19:53:41,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-23 19:53:41,998 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=72272839a6ffef279ba4e656fef71504, ASSIGN in 358 msec 2023-03-23 19:53:41,999 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:53:41,999 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601221999"}]},"ts":"1679601221999"} 2023-03-23 19:53:42,002 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-23 19:53:42,005 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:53:42,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3170 sec 2023-03-23 19:53:42,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-23 19:53:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-23 19:53:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:53:42,140 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-23 19:53:42,154 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-23 19:53:42,159 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-03-23 19:53:42,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-23 19:53:42,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-23 19:53:42,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-23 19:53:42,189 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-23 19:53:42,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-23 19:53:42,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.099sec 2023-03-23 19:53:42,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-23 19:53:42,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-23 19:53:42,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-23 19:53:42,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1679601216784-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-23 19:53:42,198 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1679601216784-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-23 19:53:42,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-23 19:53:42,264 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5e324982 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:42,268 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373f9c99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:42,282 DEBUG [hconnection-0x258148f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:42,296 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46756, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:42,306 INFO [Listener at localhost/45187] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:53:42,327 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=170, AvailableMemoryMB=6277 2023-03-23 19:53:42,343 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:53:42,346 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51282, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:53:42,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:53:42,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:53:42,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:53:42,362 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:53:42,364 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:53:42,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-23 19:53:42,367 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:42,368 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e empty. 2023-03-23 19:53:42,370 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:42,370 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:53:42,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-23 19:53:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-23 19:53:42,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-23 19:53:42,787 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:53:42,790 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 01acdc2ff8c96b2160b74ed8a9a21f8e, NAME => 'TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:53:42,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:42,802 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 01acdc2ff8c96b2160b74ed8a9a21f8e, disabling compactions & flushes 2023-03-23 19:53:42,802 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:42,803 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:42,803 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. after waiting 0 ms 2023-03-23 19:53:42,803 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:42,803 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:42,803 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:42,807 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:53:42,809 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601222809"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601222809"}]},"ts":"1679601222809"} 2023-03-23 19:53:42,812 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:53:42,813 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:53:42,813 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601222813"}]},"ts":"1679601222813"} 2023-03-23 19:53:42,816 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:53:42,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, ASSIGN}] 2023-03-23 19:53:42,823 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, ASSIGN 2023-03-23 19:53:42,825 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:53:42,977 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=01acdc2ff8c96b2160b74ed8a9a21f8e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:42,977 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601222976"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601222976"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601222976"}]},"ts":"1679601222976"} 2023-03-23 19:53:42,981 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:53:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-23 19:53:43,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 01acdc2ff8c96b2160b74ed8a9a21f8e, NAME => 'TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,143 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,145 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A 2023-03-23 19:53:43,145 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A 2023-03-23 19:53:43,146 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:53:43,147 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01acdc2ff8c96b2160b74ed8a9a21f8e columnFamilyName A 2023-03-23 19:53:43,148 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(310): Store=01acdc2ff8c96b2160b74ed8a9a21f8e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:43,148 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,149 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B 2023-03-23 19:53:43,150 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B 2023-03-23 19:53:43,150 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:53:43,150 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01acdc2ff8c96b2160b74ed8a9a21f8e columnFamilyName B 2023-03-23 19:53:43,151 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(310): Store=01acdc2ff8c96b2160b74ed8a9a21f8e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:43,151 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,152 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C 2023-03-23 19:53:43,152 DEBUG [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C 2023-03-23 19:53:43,153 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:53:43,153 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 01acdc2ff8c96b2160b74ed8a9a21f8e columnFamilyName C 2023-03-23 19:53:43,153 INFO [StoreOpener-01acdc2ff8c96b2160b74ed8a9a21f8e-1] regionserver.HStore(310): Store=01acdc2ff8c96b2160b74ed8a9a21f8e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:53:43,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:43,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:53:43,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:53:43,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 01acdc2ff8c96b2160b74ed8a9a21f8e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70205214, jitterRate=0.04613921046257019}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:53:43,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:43,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., pid=11, masterSystemTime=1679601223134 2023-03-23 19:53:43,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:43,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:43,170 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=01acdc2ff8c96b2160b74ed8a9a21f8e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,170 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601223169"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601223169"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601223169"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601223169"}]},"ts":"1679601223169"} 2023-03-23 19:53:43,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-23 19:53:43,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 in 193 msec 2023-03-23 19:53:43,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-23 19:53:43,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, ASSIGN in 354 msec 2023-03-23 19:53:43,180 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:53:43,180 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601223180"}]},"ts":"1679601223180"} 2023-03-23 19:53:43,182 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:53:43,185 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:53:43,187 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 827 msec 2023-03-23 19:53:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-23 19:53:43,489 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-23 19:53:43,494 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x0b17e1c1 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,499 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35594b1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,501 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,504 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47786, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,508 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:53:43,510 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43410, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:53:43,519 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,524 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,525 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,529 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,529 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,533 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,534 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,538 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,538 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,543 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,543 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,548 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,549 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,567 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,568 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,572 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,572 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x4bd6fc59 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:53:43,577 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4808c6d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:53:43,582 DEBUG [hconnection-0x6b1a80a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,586 DEBUG [hconnection-0x38e5ca1f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,586 DEBUG [hconnection-0x24549171-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,589 DEBUG [hconnection-0xe980b87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,589 DEBUG [hconnection-0x5457fe1e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,589 DEBUG [hconnection-0x5926a044-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,591 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,592 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47800, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,593 DEBUG [hconnection-0x536509ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,594 DEBUG [hconnection-0x6f8fa9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,594 DEBUG [hconnection-0x6462a049-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:53:43,601 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47830, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47846, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,604 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47820, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,605 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47856, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,606 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47876, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,608 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:43,613 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:53:43,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:43,627 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:43,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:43,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:43,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:43,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:43,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:43,630 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:43,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:43,634 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,634 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:43,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:43,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:43,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:43,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:43,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:43,666 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:43,667 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:43,670 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:43,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:43,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:43,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:43,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:43,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:43,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:43,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:43,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:43,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601283808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601283810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679601223753, max timestamp=1679601223808Num uniques 859; 2023-03-23 19:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-23 19:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601223753, max timestamp=1679601223808Num uniques 859; 2023-03-23 19:53:43,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:53:43,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679601223753, max timestamp=1679601223808Num uniques 859; 2023-03-23 19:53:43,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-23 19:53:43,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601283875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601283878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601283882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601283962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601283975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601283985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:43,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:43,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601283992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601283995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601284173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,192 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fb58c109a4824554bc2666f2a2b19a96 2023-03-23 19:53:44,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601284189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601284191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601284200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601284215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,346 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/295da66a285649a2b43e77ec87b3909f 2023-03-23 19:53:44,432 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/c977440f1b3d4b71ad527f49c207bdf4 2023-03-23 19:53:44,447 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fb58c109a4824554bc2666f2a2b19a96 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96 2023-03-23 19:53:44,459 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96, entries=150, sequenceid=11, filesize=11.6 K 2023-03-23 19:53:44,468 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/295da66a285649a2b43e77ec87b3909f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f 2023-03-23 19:53:44,491 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f, entries=150, sequenceid=11, filesize=11.6 K 2023-03-23 19:53:44,495 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/c977440f1b3d4b71ad527f49c207bdf4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4 2023-03-23 19:53:44,514 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4, entries=150, sequenceid=11, filesize=11.6 K 2023-03-23 19:53:44,521 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.39 KB/136590 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 851ms, sequenceid=11, compaction requested=false 2023-03-23 19:53:44,523 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-23 19:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:44,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:44,531 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:44,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3314a46[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:44,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:44,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:44,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:44,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:44,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:44,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:44,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:44,557 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:44,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:44,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:44,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:44,557 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:44,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:44,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=146.81 KB heapSize=423.70 KB 2023-03-23 19:53:44,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:44,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:44,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:44,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:44,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:44,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:44,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:44,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:44,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:44,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:44,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:44,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:44,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:44,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:44,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:44,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601284572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601284574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601284576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601284602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601284602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/597e0d2d969e4c88a68ee5ce68427063 2023-03-23 19:53:44,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:44,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:44,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8d1f27a295fc4e98bb6c52365eda5635 2023-03-23 19:53:44,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601284705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601284706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601284706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:44,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601284721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:44,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601284728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:44,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:44,737 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:44,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:44,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:44,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:44,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:44,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:44,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:44,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:44,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:44,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:44,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:44,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:44,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:44,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:44,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:44,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:44,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:44,766 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,766 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:44,766 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:44,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/831df8b0df474ae9b45f97055d2c855c 2023-03-23 19:53:44,768 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d8c798[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:44,777 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:44,777 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,779 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:44,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:44,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:44,780 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,780 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:44,784 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/597e0d2d969e4c88a68ee5ce68427063 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063 2023-03-23 19:53:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:44,797 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:44,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:44,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:44,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:44,797 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:44,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063, entries=150, sequenceid=50, filesize=11.6 K 2023-03-23 19:53:44,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8d1f27a295fc4e98bb6c52365eda5635 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635 2023-03-23 19:53:44,808 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,808 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,809 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,811 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635, entries=150, sequenceid=50, filesize=11.6 K 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,822 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:44,822 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:44,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:44,823 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:44,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:44,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/831df8b0df474ae9b45f97055d2c855c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c 2023-03-23 19:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:44,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c, entries=150, sequenceid=50, filesize=11.6 K 2023-03-23 19:53:44,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~146.81 KB/150330, heapSize ~423.66 KB/433824, currentSize=33.54 KB/34350 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 272ms, sequenceid=50, compaction requested=false 2023-03-23 19:53:44,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:44,923 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:44,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:44,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:44,928 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:44,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:44,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:44,937 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:44,937 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:44,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:44,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:44,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:44,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:44,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:44,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:44,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:44,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:44,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:44,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:44,974 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:44,974 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:44,974 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:44,974 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,987 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:44,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79da143[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:44,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:44,992 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,994 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:44,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:44,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:44,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:44,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/5e14b7cfb6004248bbc4fa3641847c69 2023-03-23 19:53:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,008 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,008 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601224976, max timestamp=1679601225002Num uniques 859; 2023-03-23 19:53:45,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:53:45,012 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601224976, max timestamp=1679601225003Num uniques 859; 2023-03-23 19:53:45,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:53:45,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679601224976, max timestamp=1679601225002Num uniques 859; 2023-03-23 19:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-23 19:53:45,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,030 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601285052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601285057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601285057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601285059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601285060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9717d1300b7546e7ad160d21e13ba11d 2023-03-23 19:53:45,131 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:45,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:45,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:45,141 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:45,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,142 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:45,142 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:45,142 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,143 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:45,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:45,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:45,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:45,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:45,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:45,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:45,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601285166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6000d987[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:45,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:45,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601285169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,174 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:45,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601285170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601285170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601285170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,182 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,183 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,204 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,256 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-23 19:53:45,258 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-23 19:53:45,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:45,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:45,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:45,321 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:45,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:45,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:45,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:45,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:45,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:45,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:45,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:45,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:45,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:45,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:45,346 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,346 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,347 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@696accbb[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:45,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:45,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:45,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:45,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:45,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,358 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,358 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,358 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,366 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,366 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,370 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,370 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,370 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601285371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,376 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601285376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601285382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601285384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:45,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601285384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,384 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,417 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,417 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,519 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:45,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:45,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/bc4729a9280643698c1b5950985ed62b 2023-03-23 19:53:45,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:45,535 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:45,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:45,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:45,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:45,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/5e14b7cfb6004248bbc4fa3641847c69 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69 2023-03-23 19:53:45,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69, entries=150, sequenceid=60, filesize=11.6 K 2023-03-23 19:53:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:45,554 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,555 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:45,555 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:45,555 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9717d1300b7546e7ad160d21e13ba11d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d 2023-03-23 19:53:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:45,560 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:45,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799e2209[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:45,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:45,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d, entries=150, sequenceid=60, filesize=11.6 K 2023-03-23 19:53:45,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:45,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/bc4729a9280643698c1b5950985ed62b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b 2023-03-23 19:53:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,588 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,589 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,594 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b, entries=150, sequenceid=60, filesize=11.6 K 2023-03-23 19:53:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,599 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 673ms, sequenceid=60, compaction requested=true 2023-03-23 19:53:45,600 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:45,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:45,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:45,603 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:45,603 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,607 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,616 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:45,619 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:45,619 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,619 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=34.8 K 2023-03-23 19:53:45,620 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,620 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:45,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:45,625 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,625 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,625 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,625 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=34.8 K 2023-03-23 19:53:45,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 295da66a285649a2b43e77ec87b3909f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679601223636 2023-03-23 19:53:45,628 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8d1f27a295fc4e98bb6c52365eda5635, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601223879 2023-03-23 19:53:45,629 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9717d1300b7546e7ad160d21e13ba11d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:45,635 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fb58c109a4824554bc2666f2a2b19a96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679601223636 2023-03-23 19:53:45,637 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 597e0d2d969e4c88a68ee5ce68427063, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601223879 2023-03-23 19:53:45,638 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5e14b7cfb6004248bbc4fa3641847c69, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:45,668 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#10 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:45,668 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#9 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:45,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=147.20 KB heapSize=424.62 KB 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:45,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:53:45,725 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:45,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:45,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601285711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601285720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601285722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:45,741 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:45,741 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:45,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:45,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:45,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:45,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601285730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601285739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:45,769 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,769 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:45,770 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:45,770 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:45,772 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:45,776 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:45,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e303c0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:45,779 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:45,779 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:45,782 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:45,783 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,783 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,789 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fd47757b61fc44549941ef6630c0411c 2023-03-23 19:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,798 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,798 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,810 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,817 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,817 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,817 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,817 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,818 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,818 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,818 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,818 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,817 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,818 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,818 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:45,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8d8013c83710439a99351bd6618813c7 2023-03-23 19:53:45,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601285841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601285844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601285845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601285868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:45,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601285868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,918 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:45,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:45,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:45,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:45,928 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:45,928 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:45,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:45,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,934 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,935 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,936 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:45,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:45,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:45,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:45,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:45,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:45,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:45,943 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75493cda[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:45,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:45,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,951 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:45,951 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:45,951 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:45,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:45,954 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:45,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:45,961 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:45,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:45,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:45,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:45,962 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:45,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:45,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:45,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:45,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:45,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:45,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:45,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:45,975 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:45,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,976 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:45,977 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:45,977 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,977 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:45,977 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:45,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:45,977 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:45,978 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:45,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:46,031 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:53:46,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601286051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601286053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601286052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601286071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601286073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,078 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:46,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:46,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:46,096 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:46,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,102 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:46,102 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:46,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,105 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,105 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:46,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:46,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:46,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:46,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:46,123 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,123 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:46,123 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:46,123 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:46,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:46,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7939dc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:46,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:46,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,133 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:46,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:46,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:46,144 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:46,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:46,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:46,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:46,145 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:46,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,155 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,156 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/f5c214a4732a4beaa5a28ea532f2995f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f5c214a4732a4beaa5a28ea532f2995f 2023-03-23 19:53:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:46,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:46,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:46,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:46,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,162 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/75e942f12ecb4ce5afc0b4b9a263b549 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/75e942f12ecb4ce5afc0b4b9a263b549 2023-03-23 19:53:46,163 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:46,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,186 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into 75e942f12ecb4ce5afc0b4b9a263b549(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:46,186 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,186 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601225602; duration=0sec 2023-03-23 19:53:46,187 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into f5c214a4732a4beaa5a28ea532f2995f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:46,187 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,187 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601225602; duration=0sec 2023-03-23 19:53:46,188 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:46,189 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:46,189 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:46,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:46,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:46,191 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,192 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=34.8 K 2023-03-23 19:53:46,194 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c977440f1b3d4b71ad527f49c207bdf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679601223636 2023-03-23 19:53:46,195 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 831df8b0df474ae9b45f97055d2c855c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601223879 2023-03-23 19:53:46,196 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bc4729a9280643698c1b5950985ed62b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:46,221 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#14 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:46,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:46,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:46,268 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/5ebc6c35531c4f8cba268e3d250a22e9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5ebc6c35531c4f8cba268e3d250a22e9 2023-03-23 19:53:46,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:46,273 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:46,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:46,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:46,278 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,278 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,280 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,281 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:46,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:46,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:46,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:46,285 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:46,285 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:46,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:46,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:46,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:46,294 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 5ebc6c35531c4f8cba268e3d250a22e9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:46,294 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,294 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601225602; duration=0sec 2023-03-23 19:53:46,294 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:46,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@427fc4f0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:46,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:46,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,301 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:46,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:46,305 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:46,312 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:46,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:46,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:46,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:46,312 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:46,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d501befee1f0413dab0ccdcfeee18fad 2023-03-23 19:53:46,316 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,316 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:46,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,330 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:46,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:46,332 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:46,332 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:46,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fd47757b61fc44549941ef6630c0411c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c 2023-03-23 19:53:46,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c, entries=250, sequenceid=99, filesize=16.2 K 2023-03-23 19:53:46,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8d8013c83710439a99351bd6618813c7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7 2023-03-23 19:53:46,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601286364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601286364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601286366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7, entries=150, sequenceid=99, filesize=11.6 K 2023-03-23 19:53:46,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d501befee1f0413dab0ccdcfeee18fad as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad 2023-03-23 19:53:46,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601286379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601286379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad, entries=150, sequenceid=99, filesize=11.6 K 2023-03-23 19:53:46,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.20 KB/150735, heapSize ~424.57 KB/434760, currentSize=33.54 KB/34350 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 705ms, sequenceid=99, compaction requested=false 2023-03-23 19:53:46,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,432 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:46,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:46,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:46,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:46,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:46,438 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:46,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:46,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:46,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,447 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:46,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:46,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:46,452 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:46,453 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,501 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/53276d16997f44f0842dfb0a0878fae5 2023-03-23 19:53:46,534 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ac0049dfc1c345a199ec49bdab6d17f5 2023-03-23 19:53:46,565 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/e1edfeed451540e3b69e8212c9360b74 2023-03-23 19:53:46,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/53276d16997f44f0842dfb0a0878fae5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5 2023-03-23 19:53:46,587 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5, entries=100, sequenceid=110, filesize=9.3 K 2023-03-23 19:53:46,589 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ac0049dfc1c345a199ec49bdab6d17f5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5 2023-03-23 19:53:46,598 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5, entries=100, sequenceid=110, filesize=9.3 K 2023-03-23 19:53:46,601 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/e1edfeed451540e3b69e8212c9360b74 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74 2023-03-23 19:53:46,618 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74, entries=100, sequenceid=110, filesize=9.3 K 2023-03-23 19:53:46,620 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 168ms, sequenceid=110, compaction requested=true 2023-03-23 19:53:46,620 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,620 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:46,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:46,627 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4909327c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:46,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:46,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:46,637 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,637 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:46,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:46,647 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:46,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:46,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:46,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:46,648 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:46,651 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,651 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,651 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,653 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,658 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,663 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,663 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:46,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:46,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:46,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:46,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:46,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:46,773 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:46,773 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,776 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:46,776 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:46,777 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,777 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,780 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,780 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,781 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,781 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:46,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:46,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:46,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:46,789 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,789 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:46,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:46,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:46,791 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:46,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,797 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:46,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751a686b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:46,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:46,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:46,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:46,808 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,808 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:46,811 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:46,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:46,826 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:46,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:46,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:46,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:46,826 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:46,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:46,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:46,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:46,843 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:46,843 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:46,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:46,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:46,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:46,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:46,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:46,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/55474feaa4b6454ab4133caf04cee0d6 2023-03-23 19:53:46,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:46,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:46,954 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:46,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:46,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:46,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,957 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-23 19:53:46,958 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-23 19:53:46,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:46,958 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:46,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:46,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:46,973 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601226915, max timestamp=1679601226967Num uniques 859; 2023-03-23 19:53:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601226915, max timestamp=1679601226967Num uniques 859; 2023-03-23 19:53:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:53:46,975 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:46,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:53:46,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:46,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:46,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:46,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:46,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:46,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ae08cd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:46,990 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:46,990 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,993 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:46,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:46,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:47,002 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:47,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:47,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:47,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:47,002 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:47,006 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,006 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,006 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601286998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601287003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,016 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:47,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:47,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:47,016 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,018 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:47,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601287112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,118 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:47,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:47,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:47,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601287119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:47,129 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:47,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:47,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:47,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:47,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:47,149 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:47,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:47,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:47,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:47,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:47,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:47,154 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:47,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a4a1aac[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:47,158 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:47,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:47,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:47,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:47,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:47,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:47,168 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:47,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:47,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:47,169 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:47,169 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:47,170 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,171 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,171 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:47,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:47,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:47,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:47,183 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:47,183 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:47,283 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:47,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:47,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:47,289 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:47,289 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:47,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:47,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,293 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:47,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:47,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:47,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:47,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:47,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:47,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:47,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:47,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:47,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:47,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a504b4b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:47,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:47,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:47,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:47,318 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:47,324 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:47,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:47,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:47,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:47,325 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:47,327 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,327 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601287324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:47,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:47,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,337 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601287334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:47,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:47,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:47,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:47,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:47,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,345 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:47,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:47,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:47,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9e3287b4bae7478eafd19857f74fb74a 2023-03-23 19:53:47,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/43d891909a4e40feb0c7b910a8e23831 2023-03-23 19:53:47,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/55474feaa4b6454ab4133caf04cee0d6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6 2023-03-23 19:53:47,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6, entries=200, sequenceid=121, filesize=13.9 K 2023-03-23 19:53:47,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9e3287b4bae7478eafd19857f74fb74a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a 2023-03-23 19:53:47,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a, entries=150, sequenceid=121, filesize=11.6 K 2023-03-23 19:53:47,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/43d891909a4e40feb0c7b910a8e23831 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831 2023-03-23 19:53:47,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831, entries=150, sequenceid=121, filesize=11.6 K 2023-03-23 19:53:47,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 555ms, sequenceid=121, compaction requested=true 2023-03-23 19:53:47,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:47,441 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:47,441 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:47,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:47,442 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:47,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:47,448 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52323 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:47,448 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:47,448 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:47,449 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:47,449 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,449 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,449 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f5c214a4732a4beaa5a28ea532f2995f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=51.1 K 2023-03-23 19:53:47,449 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/75e942f12ecb4ce5afc0b4b9a263b549, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=44.2 K 2023-03-23 19:53:47,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:47,450 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f5c214a4732a4beaa5a28ea532f2995f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:47,450 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 75e942f12ecb4ce5afc0b4b9a263b549, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:47,452 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting fd47757b61fc44549941ef6630c0411c, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679601225048 2023-03-23 19:53:47,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:47,453 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 53276d16997f44f0842dfb0a0878fae5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601225721 2023-03-23 19:53:47,454 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 8d8013c83710439a99351bd6618813c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679601225058 2023-03-23 19:53:47,454 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 55474feaa4b6454ab4133caf04cee0d6, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:47,454 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ac0049dfc1c345a199ec49bdab6d17f5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601225721 2023-03-23 19:53:47,456 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9e3287b4bae7478eafd19857f74fb74a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:47,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:47,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:47,459 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:47,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:47,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:47,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:47,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:47,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:47,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:47,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:47,474 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,474 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:47,474 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=131.94 KB heapSize=383.31 KB 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:47,494 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:47,502 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:47,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:47,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601287641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601287641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601287749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601287955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601287956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601287957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601287958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:47,973 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/e2b15a2ba6f94cd3a9c045b9551ec3a0 2023-03-23 19:53:47,980 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/be3d252b16a54e8bac2d46ac1232c1f1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/be3d252b16a54e8bac2d46ac1232c1f1 2023-03-23 19:53:47,985 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/dbf96f9bc1c949468bd53f9f48fdd8aa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/dbf96f9bc1c949468bd53f9f48fdd8aa 2023-03-23 19:53:47,994 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into be3d252b16a54e8bac2d46ac1232c1f1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:47,994 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:47,994 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=12, startTime=1679601227441; duration=0sec 2023-03-23 19:53:47,995 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:47,995 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:47,998 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:47,998 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:47,998 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into dbf96f9bc1c949468bd53f9f48fdd8aa(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:47,998 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:47,999 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5ebc6c35531c4f8cba268e3d250a22e9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=44.2 K 2023-03-23 19:53:47,999 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5ebc6c35531c4f8cba268e3d250a22e9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679601224587 2023-03-23 19:53:47,998 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:48,002 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=12, startTime=1679601227442; duration=0sec 2023-03-23 19:53:48,000 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d501befee1f0413dab0ccdcfeee18fad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679601225058 2023-03-23 19:53:48,002 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:48,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e1edfeed451540e3b69e8212c9360b74, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601225721 2023-03-23 19:53:48,012 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 43d891909a4e40feb0c7b910a8e23831, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:48,026 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:48,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601288150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601288258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601288259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601288262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601288262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,414 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/02960f24372047a796c77323b23c5c00 2023-03-23 19:53:48,451 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/7c67a6d6f0d142709d92c94ae0ccbc10 2023-03-23 19:53:48,458 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a22626c3c04b43368b0927f4a231ba55 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a22626c3c04b43368b0927f4a231ba55 2023-03-23 19:53:48,470 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into a22626c3c04b43368b0927f4a231ba55(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:48,470 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:48,470 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=12, startTime=1679601227442; duration=0sec 2023-03-23 19:53:48,471 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:48,471 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/e2b15a2ba6f94cd3a9c045b9551ec3a0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0 2023-03-23 19:53:48,487 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0, entries=150, sequenceid=153, filesize=11.8 K 2023-03-23 19:53:48,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/02960f24372047a796c77323b23c5c00 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00 2023-03-23 19:53:48,504 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00, entries=150, sequenceid=153, filesize=11.8 K 2023-03-23 19:53:48,509 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/7c67a6d6f0d142709d92c94ae0ccbc10 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10 2023-03-23 19:53:48,518 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10, entries=150, sequenceid=153, filesize=11.8 K 2023-03-23 19:53:48,533 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~383.27 KB/392464, currentSize=46.96 KB/48090 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 1059ms, sequenceid=153, compaction requested=false 2023-03-23 19:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:48,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:48,537 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:48,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6caf2b5a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:48,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:48,542 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:48,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:48,544 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,544 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:48,546 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:48,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:48,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:48,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:48,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:48,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:48,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:48,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:48,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:48,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:48,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:48,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:48,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:48,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:48,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:48,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:48,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:48,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:48,679 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:48,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:48,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:48,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:48,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:48,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:48,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:48,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:48,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:48,687 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:48,687 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:48,687 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:48,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:48,724 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/20f46d5b285e4efcaf3916c42d497719 2023-03-23 19:53:48,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:48,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:48,775 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/72084c8ff7f54c568b7098e84dbbcf0b 2023-03-23 19:53:48,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601288794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679601228781, max timestamp=1679601228797Num uniques 859; 2023-03-23 19:53:48,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-23 19:53:48,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679601228781, max timestamp=1679601228797Num uniques 859; 2023-03-23 19:53:48,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679601228781, max timestamp=1679601228797Num uniques 859; 2023-03-23 19:53:48,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-23 19:53:48,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-23 19:53:48,803 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/ae7c30c37d4a451bb0cd85005403df93 2023-03-23 19:53:48,811 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/20f46d5b285e4efcaf3916c42d497719 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719 2023-03-23 19:53:48,820 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719, entries=150, sequenceid=166, filesize=11.8 K 2023-03-23 19:53:48,826 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/72084c8ff7f54c568b7098e84dbbcf0b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b 2023-03-23 19:53:48,833 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b, entries=150, sequenceid=166, filesize=11.8 K 2023-03-23 19:53:48,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/ae7c30c37d4a451bb0cd85005403df93 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93 2023-03-23 19:53:48,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601288831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601288833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601288833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,845 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93, entries=150, sequenceid=166, filesize=11.8 K 2023-03-23 19:53:48,847 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 160ms, sequenceid=166, compaction requested=true 2023-03-23 19:53:48,847 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:48,847 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:48,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a41be62[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:48,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:48,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:48,862 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:48,868 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:48,868 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:48,868 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:48,868 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:48,868 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:48,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,872 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:48,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:48,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:48,877 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:48,877 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:48,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:48,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-03-23 19:53:48,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:48,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:48,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:48,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:48,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:48,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:48,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/880977e8338644bf9e485298e2cb43ed 2023-03-23 19:53:48,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601288932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601288943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601288944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601288944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:48,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:48,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:48,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:48,984 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:48,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:48,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:48,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:48,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:48,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:48,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:48,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:48,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:48,991 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:48,994 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:48,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:48,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:48,996 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:48,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704446fc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:48,996 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,998 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:48,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:48,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:48,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,003 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,003 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,009 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,009 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,009 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:49,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601289039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,122 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@388651ac[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,137 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,141 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,141 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601289147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601289150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,157 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601289149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:49,157 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,158 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,158 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,161 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601289161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601289245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,262 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,267 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,267 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,268 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,268 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,269 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,269 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,270 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,270 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,275 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,275 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,275 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,277 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de6481d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,282 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,283 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,283 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,283 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,288 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,288 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,296 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,296 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,296 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/12d1d957b28c4bada42ceca816e6abc0 2023-03-23 19:53:49,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a6b859e33f8d4dfcb3a8ae08dd551d19 2023-03-23 19:53:49,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/880977e8338644bf9e485298e2cb43ed as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed 2023-03-23 19:53:49,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,412 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,414 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed, entries=200, sequenceid=202, filesize=14.1 K 2023-03-23 19:53:49,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/12d1d957b28c4bada42ceca816e6abc0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0 2023-03-23 19:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ab3516d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,429 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,429 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,438 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,438 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0, entries=150, sequenceid=202, filesize=11.8 K 2023-03-23 19:53:49,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a6b859e33f8d4dfcb3a8ae08dd551d19 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:49,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,451 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19, entries=150, sequenceid=202, filesize=11.8 K 2023-03-23 19:53:49,461 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 555ms, sequenceid=202, compaction requested=true 2023-03-23 19:53:49,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:49,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:49,461 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:49,461 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:49,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:49,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:49,464 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:49,464 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50624 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:49,464 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:49,464 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:49,464 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,464 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,464 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/dbf96f9bc1c949468bd53f9f48fdd8aa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=47.1 K 2023-03-23 19:53:49,464 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/be3d252b16a54e8bac2d46ac1232c1f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=49.4 K 2023-03-23 19:53:49,465 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting dbf96f9bc1c949468bd53f9f48fdd8aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:49,465 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting be3d252b16a54e8bac2d46ac1232c1f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:49,467 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 02960f24372047a796c77323b23c5c00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679601226985 2023-03-23 19:53:49,467 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e2b15a2ba6f94cd3a9c045b9551ec3a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679601226985 2023-03-23 19:53:49,467 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 72084c8ff7f54c568b7098e84dbbcf0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1679601227638 2023-03-23 19:53:49,467 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 20f46d5b285e4efcaf3916c42d497719, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1679601227638 2023-03-23 19:53:49,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 12d1d957b28c4bada42ceca816e6abc0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:49,470 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 880977e8338644bf9e485298e2cb43ed, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:49,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:49,471 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:49,471 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:49,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:49,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:49,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:49,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:49,505 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:49,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/324522bd8f564510ad269aa30a17fd12 2023-03-23 19:53:49,512 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601229495, max timestamp=1679601229524Num uniques 859; 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601229495, max timestamp=1679601229525Num uniques 859; 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601229495, max timestamp=1679601229525Num uniques 859; 2023-03-23 19:53:49,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:49,545 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b48dd99ef3064b08b29cd49338d31faa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b48dd99ef3064b08b29cd49338d31faa 2023-03-23 19:53:49,551 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,558 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into b48dd99ef3064b08b29cd49338d31faa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:49,558 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:49,558 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=12, startTime=1679601229461; duration=0sec 2023-03-23 19:53:49,558 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:49,559 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:53:49,561 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:53:49,561 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:49,561 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,561 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a22626c3c04b43368b0927f4a231ba55, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=47.1 K 2023-03-23 19:53:49,562 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a22626c3c04b43368b0927f4a231ba55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679601226871 2023-03-23 19:53:49,562 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7c67a6d6f0d142709d92c94ae0ccbc10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679601226985 2023-03-23 19:53:49,563 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ae7c30c37d4a451bb0cd85005403df93, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1679601227638 2023-03-23 19:53:49,564 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a6b859e33f8d4dfcb3a8ae08dd551d19, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,577 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/241ea27918bd408fb3f2953183d31f81 2023-03-23 19:53:49,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601289563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601289566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601289579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601289581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,602 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18929d61[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,606 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,606 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,607 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,610 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,616 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,616 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,616 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,616 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,616 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,623 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,626 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,626 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,627 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,627 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a3fcec2a180441478f222c13d7da3c29 2023-03-23 19:53:49,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/324522bd8f564510ad269aa30a17fd12 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12 2023-03-23 19:53:49,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12, entries=250, sequenceid=213, filesize=16.4 K 2023-03-23 19:53:49,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/241ea27918bd408fb3f2953183d31f81 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81 2023-03-23 19:53:49,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81, entries=150, sequenceid=213, filesize=11.8 K 2023-03-23 19:53:49,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a3fcec2a180441478f222c13d7da3c29 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29 2023-03-23 19:53:49,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29, entries=150, sequenceid=213, filesize=11.8 K 2023-03-23 19:53:49,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 209ms, sequenceid=213, compaction requested=false 2023-03-23 19:53:49,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:49,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:49,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:49,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601289693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/11437c9b93d3464691ca2cb4b10e8c86 2023-03-23 19:53:49,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601289700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601289702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601289706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ab53435bfb8d4fc6940c24f9884170e2 2023-03-23 19:53:49,727 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,736 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,738 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,741 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,741 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,741 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,741 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,742 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1213b4ee[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,746 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,751 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,751 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,753 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,753 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,753 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,755 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,758 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,758 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601289806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601289807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601289809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601289809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,858 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:49,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:49,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:49,864 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:49,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:49,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:49,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,872 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:49,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:49,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:49,874 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,874 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:49,874 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:49,874 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:49,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,875 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:49,875 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,876 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:49,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@daab011[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:49,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:49,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:49,885 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:49,889 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:49,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:49,897 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:49,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:49,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:49,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:49,897 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:49,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:49,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:49,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,900 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,902 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:49,908 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:49,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:49,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:49,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:49,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:49,909 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:49,909 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:49,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:49,950 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/f7ff8eb7c0324a1891c9ecc16f258d50 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f7ff8eb7c0324a1891c9ecc16f258d50 2023-03-23 19:53:49,962 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into f7ff8eb7c0324a1891c9ecc16f258d50(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:49,962 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:49,962 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=12, startTime=1679601229461; duration=0sec 2023-03-23 19:53:49,962 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:50,009 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601290010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601290010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,014 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,016 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,017 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,017 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601290015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601290016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,019 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc2b0f0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,023 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,023 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,030 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,030 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,030 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,030 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,030 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,032 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,033 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,033 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,041 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d724e8476b364a3abf1beea9539de2e6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d724e8476b364a3abf1beea9539de2e6 2023-03-23 19:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,053 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,053 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,053 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,054 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into d724e8476b364a3abf1beea9539de2e6(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:50,054 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:50,054 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=12, startTime=1679601229462; duration=0sec 2023-03-23 19:53:50,054 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:50,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/1a711dfd66014405b5d46753a9296244 2023-03-23 19:53:50,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/11437c9b93d3464691ca2cb4b10e8c86 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86 2023-03-23 19:53:50,154 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86, entries=200, sequenceid=251, filesize=14.1 K 2023-03-23 19:53:50,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ab53435bfb8d4fc6940c24f9884170e2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2 2023-03-23 19:53:50,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2, entries=150, sequenceid=251, filesize=11.8 K 2023-03-23 19:53:50,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/1a711dfd66014405b5d46753a9296244 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244 2023-03-23 19:53:50,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,173 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244, entries=150, sequenceid=251, filesize=11.8 K 2023-03-23 19:53:50,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12adf9a7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 505ms, sequenceid=251, compaction requested=true 2023-03-23 19:53:50,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:50,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,191 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:50,191 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:50,191 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:50,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:50,191 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:50,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,192 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,193 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:50,193 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,193 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,193 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,193 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b48dd99ef3064b08b29cd49338d31faa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=42.6 K 2023-03-23 19:53:50,193 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:50,193 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:50,193 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,193 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f7ff8eb7c0324a1891c9ecc16f258d50, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=35.6 K 2023-03-23 19:53:50,193 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b48dd99ef3064b08b29cd49338d31faa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:50,194 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,195 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,195 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f7ff8eb7c0324a1891c9ecc16f258d50, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:50,195 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,195 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,195 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 324522bd8f564510ad269aa30a17fd12, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679601228913 2023-03-23 19:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,196 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 241ea27918bd408fb3f2953183d31f81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679601228913 2023-03-23 19:53:50,196 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 11437c9b93d3464691ca2cb4b10e8c86, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,196 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ab53435bfb8d4fc6940c24f9884170e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,201 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,201 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,208 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:50,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,211 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:50,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,214 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,214 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,230 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/76fb3e0644574910b0f23c6435280ea1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/76fb3e0644574910b0f23c6435280ea1 2023-03-23 19:53:50,231 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/bd6ccc5856454524a440b77047cb1709 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/bd6ccc5856454524a440b77047cb1709 2023-03-23 19:53:50,238 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into 76fb3e0644574910b0f23c6435280ea1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:50,238 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:50,238 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601230191; duration=0sec 2023-03-23 19:53:50,238 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:50,238 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:50,239 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into bd6ccc5856454524a440b77047cb1709(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:50,239 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:50,239 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601230191; duration=0sec 2023-03-23 19:53:50,239 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:50,239 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:50,239 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:50,240 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,240 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d724e8476b364a3abf1beea9539de2e6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=35.6 K 2023-03-23 19:53:50,240 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d724e8476b364a3abf1beea9539de2e6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679601228829 2023-03-23 19:53:50,241 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a3fcec2a180441478f222c13d7da3c29, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679601228913 2023-03-23 19:53:50,246 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 1a711dfd66014405b5d46753a9296244, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:50,254 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#44 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:50,266 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/05d8c51ce420459c9fcc13d10d0fb6ae as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/05d8c51ce420459c9fcc13d10d0fb6ae 2023-03-23 19:53:50,273 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 05d8c51ce420459c9fcc13d10d0fb6ae(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:50,273 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:50,273 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601230191; duration=0sec 2023-03-23 19:53:50,273 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:50,314 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:50,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,320 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,328 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,328 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,328 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,328 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,334 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/03d027075f2c4d4983ac35452a0c88c4 2023-03-23 19:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b757bc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601290337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601290338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601290342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601290342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,346 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,347 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,348 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,348 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:50,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,357 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601290443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601290448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601290448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601290448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,457 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,461 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,467 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,471 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e8c2da0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,474 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,475 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,475 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,477 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,480 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,481 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,483 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,484 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,487 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,487 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,487 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,487 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,487 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:50,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,593 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,597 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,597 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,597 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,597 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe2458[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,609 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,609 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,610 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,614 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,614 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,614 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,616 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,616 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,616 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:50,623 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,623 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601290647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601290651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601290651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601290651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,723 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,730 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,731 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,731 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,732 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,732 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,732 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,732 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,734 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,734 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,734 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,734 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,735 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,735 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,737 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,737 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,737 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef34147[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,740 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,741 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,741 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,741 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,746 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,746 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:50,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/60962d12e20f4758aaaa6bcaa20bcbf1 2023-03-23 19:53:50,855 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:50,860 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:50,860 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,860 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:50,860 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:50,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,862 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,863 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,863 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:50,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:50,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:50,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:50,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:50,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:50,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:50,865 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:50,865 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:50,867 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:50,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2512e592[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:50,869 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:50,869 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:50,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:50,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:50,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:50,877 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:50,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:50,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:50,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:50,877 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:50,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:50,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:50,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:50,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:50,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:50,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:50,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:50,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:50,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601290950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601290957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601290960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601290968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:50,986 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:50,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:50,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:51,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:51,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:51,000 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:51,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:51,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:51,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:51,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:51,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,004 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:51,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:51,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:51,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:51,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:51,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:51,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:51,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:51,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:51,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a4c2dbd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:51,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:51,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:51,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:51,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:51,033 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,033 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:51,033 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,033 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:51,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:51,036 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:51,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:51,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:51,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:51,036 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:51,039 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,039 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,039 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:51,040 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,042 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:51,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:51,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,046 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:51,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:51,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:51,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,046 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:51,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,046 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:51,146 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:51,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:51,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:51,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:51,151 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:51,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:51,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:51,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:51,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:51,154 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,154 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:51,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:51,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:51,156 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:51,156 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:51,156 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:51,156 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:51,158 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:51,161 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de70648[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:51,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:51,165 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,165 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:51,165 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:51,165 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:51,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:51,167 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d185eed418e34696a9a8992cf1410389 2023-03-23 19:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:51,172 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:51,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:51,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:51,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:51,172 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:51,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:51,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:51,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:51,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/03d027075f2c4d4983ac35452a0c88c4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4 2023-03-23 19:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:51,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:51,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:51,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:51,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:51,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:51,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:51,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:51,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:51,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:51,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4, entries=150, sequenceid=268, filesize=11.9 K 2023-03-23 19:53:51,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601291180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/60962d12e20f4758aaaa6bcaa20bcbf1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1 2023-03-23 19:53:51,189 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:51,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1, entries=150, sequenceid=268, filesize=11.9 K 2023-03-23 19:53:51,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d185eed418e34696a9a8992cf1410389 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389 2023-03-23 19:53:51,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389, entries=150, sequenceid=268, filesize=11.9 K 2023-03-23 19:53:51,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 883ms, sequenceid=268, compaction requested=false 2023-03-23 19:53:51,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:51,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:51,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:51,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:51,291 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:51,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:51,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:51,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,294 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:51,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:51,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:51,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:51,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:51,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:51,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:51,297 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:51,297 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:51,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:51,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:51,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601291470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601291470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601291471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601291473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601291575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601291575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601291576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,719 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/18f9a575669f4963a63889652ee338a5 2023-03-23 19:53:51,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47872 deadline: 1679601291779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601291779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:51,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:51,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47830 deadline: 1679601291780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601292083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601292084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601292085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,146 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/429d10784102417499adea11c3dabd2b 2023-03-23 19:53:52,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601292483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,566 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/5313a75d9ae947c99ec1f7dbf1695f1f 2023-03-23 19:53:52,574 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/18f9a575669f4963a63889652ee338a5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5 2023-03-23 19:53:52,583 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5, entries=150, sequenceid=288, filesize=11.9 K 2023-03-23 19:53:52,584 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/429d10784102417499adea11c3dabd2b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b 2023-03-23 19:53:52,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47804 deadline: 1679601292586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601292590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:52,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601292591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,595 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b, entries=150, sequenceid=288, filesize=11.9 K 2023-03-23 19:53:52,596 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/5313a75d9ae947c99ec1f7dbf1695f1f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f 2023-03-23 19:53:52,602 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f, entries=150, sequenceid=288, filesize=11.9 K 2023-03-23 19:53:52,603 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 1307ms, sequenceid=288, compaction requested=true 2023-03-23 19:53:52,603 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:52,603 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,607 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394a9bca[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:52,609 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:52,609 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,611 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:52,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:52,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:52,616 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:52,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:52,628 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:52,629 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:52,629 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:52,629 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:52,629 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:52,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:52,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:52,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:52,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,636 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,640 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,640 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:52,640 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:52,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:52,641 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:52,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,641 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:52,641 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:52,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:52,741 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:52,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:52,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:52,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:52,750 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:52,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:52,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:52,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:52,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:52,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:52,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:52,753 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:52,753 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:52,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:52,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:52,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:52,755 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:52,755 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:52,767 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/9f2e68fcd6fa4d3a86a636d7c1a4d378 2023-03-23 19:53:53,184 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ef53be9d53b347b08d2cad989ea5ae47 2023-03-23 19:53:53,212 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/90840402fee8401583a299b515c5a543 2023-03-23 19:53:53,220 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/9f2e68fcd6fa4d3a86a636d7c1a4d378 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378 2023-03-23 19:53:53,233 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378, entries=150, sequenceid=300, filesize=11.9 K 2023-03-23 19:53:53,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/ef53be9d53b347b08d2cad989ea5ae47 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47 2023-03-23 19:53:53,249 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47, entries=150, sequenceid=300, filesize=11.9 K 2023-03-23 19:53:53,252 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/90840402fee8401583a299b515c5a543 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543 2023-03-23 19:53:53,260 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543, entries=150, sequenceid=300, filesize=11.9 K 2023-03-23 19:53:53,262 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 507ms, sequenceid=300, compaction requested=true 2023-03-23 19:53:53,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:53,267 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc6c704[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:53,270 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:53,270 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,272 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:53,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:53,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:53,277 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:53,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:53,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:53,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:53,277 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:53,279 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,279 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,279 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,289 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,289 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,289 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:53,289 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:53,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:53,289 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:53,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:53,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:53,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:53,404 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:53,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:53,405 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,405 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,406 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:53,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:53,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:53,414 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:53,414 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:53,414 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,414 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:53,415 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,415 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:53,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,418 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:53,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@701c2204[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:53,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:53,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,423 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:53,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:53,428 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:53,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:53,433 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:53,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:53,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:53,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:53,433 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:53,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,439 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,442 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,442 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:53,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:53,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,443 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:53,443 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,543 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:53,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:53,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:53,548 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:53,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:53,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:53,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:53,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:53,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:53,580 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,580 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:53,580 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,580 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:53,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:53,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:53,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:53,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251caf09[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:53,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:53,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:53,590 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,590 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:53,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:53,595 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:53,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:53,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:53,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:53,595 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:53,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,603 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,613 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:53,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:53,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:53,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fcb62e5c2834404cbf6b5c1f29b14f47 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601233646, max timestamp=1679601233662Num uniques 859; 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601233646, max timestamp=1679601233661Num uniques 859; 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601233646, max timestamp=1679601233662Num uniques 859; 2023-03-23 19:53:53,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:53,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601293699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601293703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601293697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:53,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/eaed619f79ef463d823e993f771a69ed 2023-03-23 19:53:53,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:53,720 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:53,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:53,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:53,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,723 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,724 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,724 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:53,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:53,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:53,744 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,744 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:53,744 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:53,744 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:53,747 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,747 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4865f4ab[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:53,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:53,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:53,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:53,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:53,762 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a67c66bce1f44bba8365dd1c1ca0dba8 2023-03-23 19:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:53,768 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:53,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:53,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:53,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:53,768 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:53,779 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,779 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,779 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:53,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fcb62e5c2834404cbf6b5c1f29b14f47 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47 2023-03-23 19:53:53,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,780 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,790 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,790 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:53,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:53,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:53,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:53,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:53,790 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,790 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:53,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:53,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:53,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:53,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47, entries=250, sequenceid=312, filesize=16.7 K 2023-03-23 19:53:53,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/eaed619f79ef463d823e993f771a69ed as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed 2023-03-23 19:53:53,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed, entries=100, sequenceid=312, filesize=9.5 K 2023-03-23 19:53:53,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601293804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a67c66bce1f44bba8365dd1c1ca0dba8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8 2023-03-23 19:53:53,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601293806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601293807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8, entries=100, sequenceid=312, filesize=9.5 K 2023-03-23 19:53:53,818 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 204ms, sequenceid=312, compaction requested=true 2023-03-23 19:53:53,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:53,819 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:53,819 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:53,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:53,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:53,821 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58764 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:53,821 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:53,821 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,821 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66084 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:53,821 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/76fb3e0644574910b0f23c6435280ea1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=57.4 K 2023-03-23 19:53:53,821 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:53,822 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,822 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/bd6ccc5856454524a440b77047cb1709, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=64.5 K 2023-03-23 19:53:53,822 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 76fb3e0644574910b0f23c6435280ea1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:53,824 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting bd6ccc5856454524a440b77047cb1709, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:53,828 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 60962d12e20f4758aaaa6bcaa20bcbf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679601229689 2023-03-23 19:53:53,828 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 03d027075f2c4d4983ac35452a0c88c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679601229689 2023-03-23 19:53:53,828 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 429d10784102417499adea11c3dabd2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679601230331 2023-03-23 19:53:53,829 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ef53be9d53b347b08d2cad989ea5ae47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679601231468 2023-03-23 19:53:53,829 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 18f9a575669f4963a63889652ee338a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679601230331 2023-03-23 19:53:53,829 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting eaed619f79ef463d823e993f771a69ed, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601233609 2023-03-23 19:53:53,829 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9f2e68fcd6fa4d3a86a636d7c1a4d378, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679601231468 2023-03-23 19:53:53,831 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fcb62e5c2834404cbf6b5c1f29b14f47, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601233598 2023-03-23 19:53:53,878 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:53,891 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#58 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:53,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:53,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:53,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:53,909 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:53,909 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,912 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:53,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:53,913 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:53,913 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:53,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:53,942 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ec0b4d5b077e421483b539f10e4244b4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec0b4d5b077e421483b539f10e4244b4 2023-03-23 19:53:53,950 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into ec0b4d5b077e421483b539f10e4244b4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:53,950 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,950 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=11, startTime=1679601233819; duration=0sec 2023-03-23 19:53:53,950 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:53,950 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:53,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:53,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:53,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:53,954 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58764 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:53,954 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:53,955 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,955 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/05d8c51ce420459c9fcc13d10d0fb6ae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=57.4 K 2023-03-23 19:53:53,955 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 05d8c51ce420459c9fcc13d10d0fb6ae, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601229563 2023-03-23 19:53:53,956 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d185eed418e34696a9a8992cf1410389, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679601229689 2023-03-23 19:53:53,956 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5313a75d9ae947c99ec1f7dbf1695f1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679601230331 2023-03-23 19:53:53,957 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 90840402fee8401583a299b515c5a543, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679601231468 2023-03-23 19:53:53,957 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a67c66bce1f44bba8365dd1c1ca0dba8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601233609 2023-03-23 19:53:53,959 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:53,959 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:53,959 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:53,959 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:53,959 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:53,970 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6ae6be1d09204a919223b3c3a68f8e16 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6ae6be1d09204a919223b3c3a68f8e16 2023-03-23 19:53:53,983 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#60 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:53,989 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into 6ae6be1d09204a919223b3c3a68f8e16(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:53,989 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:53,989 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=11, startTime=1679601233819; duration=0sec 2023-03-23 19:53:53,989 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:54,014 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ef1ddb04a2784942811437233761461f 2023-03-23 19:53:54,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:54,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:54,027 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/781c15e1e6934a8a913fd60251581960 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/781c15e1e6934a8a913fd60251581960 2023-03-23 19:53:54,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601294025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601294023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601294027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,039 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 781c15e1e6934a8a913fd60251581960(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:54,040 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:54,040 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=11, startTime=1679601233819; duration=0sec 2023-03-23 19:53:54,040 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:54,045 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/e6a8fdfb9a3d4a1e934e0485273bc8a4 2023-03-23 19:53:54,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601294131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601294132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601294137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601294334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601294336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601294340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,473 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a142c37b18f44a2b83d6d42794ff9586 2023-03-23 19:53:54,480 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ef1ddb04a2784942811437233761461f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f 2023-03-23 19:53:54,485 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f, entries=150, sequenceid=349, filesize=11.9 K 2023-03-23 19:53:54,486 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/e6a8fdfb9a3d4a1e934e0485273bc8a4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4 2023-03-23 19:53:54,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601294488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,508 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:54,511 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4, entries=150, sequenceid=349, filesize=11.9 K 2023-03-23 19:53:54,512 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/a142c37b18f44a2b83d6d42794ff9586 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586 2023-03-23 19:53:54,518 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586, entries=150, sequenceid=349, filesize=11.9 K 2023-03-23 19:53:54,519 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 560ms, sequenceid=349, compaction requested=false 2023-03-23 19:53:54,519 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:54,519 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:54,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a6f444[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:54,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:54,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,525 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:54,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:54,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:54,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:54,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:54,531 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:54,531 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:54,531 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:54,531 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:54,531 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:54,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,535 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:54,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:54,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,544 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:54,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,544 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:54,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:54,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:53:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:54,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:54,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:54,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:54,644 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:54,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:54,648 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,651 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:54,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:54,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:54,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:54,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:54,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:54,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:54,672 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:54,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601234665, max timestamp=1679601234673Num uniques 859; 2023-03-23 19:53:54,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:54,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601234665, max timestamp=1679601234673Num uniques 859; 2023-03-23 19:53:54,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601234665, max timestamp=1679601234673Num uniques 859; 2023-03-23 19:53:54,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64bbe4f6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:54,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:54,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:54,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:54,681 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:54,684 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:54,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:54,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:54,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:54,684 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:54,686 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,686 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,686 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:54,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:54,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,695 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,696 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601294695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601294696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601294696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,796 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:54,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:54,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:54,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601294800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:54,804 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:54,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:54,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:54,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601294802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601294802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,808 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:54,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:54,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:54,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:54,810 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,810 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:54,810 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:54,810 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:54,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:54,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241f7b8f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:54,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:54,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,816 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:54,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:54,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:54,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:54,818 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:54,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:54,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:54,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:54,819 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:54,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,822 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,825 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:54,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:54,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:54,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:54,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:54,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:54,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:54,931 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:54,931 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,931 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:54,931 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:54,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,933 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:54,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:54,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:54,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:54,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:54,940 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:54,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:54,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,943 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:54,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4338130b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:54,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:54,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,948 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:54,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:54,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:54,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:54,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:54,953 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:54,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:54,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:54,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:54,953 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:54,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:54,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:54,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:54,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:54,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:54,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:54,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:54,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:54,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:54,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:55,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601295002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601295008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601295008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:55,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:55,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/e59c11e8f8e942fd87394d4bb51f2139 2023-03-23 19:53:55,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:55,065 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:55,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:55,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:55,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,074 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:55,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:55,085 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,085 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:55,085 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:55,085 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:55,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/5dba76e04bb941a880c41a9bb10614a1 2023-03-23 19:53:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:55,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fedbfc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:55,098 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:55,098 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,099 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:55,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:55,101 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:55,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:55,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:55,107 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:55,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:55,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:55,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:55,107 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:55,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,111 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,112 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:55,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:55,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,115 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:55,115 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:55,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47846 deadline: 1679601295210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,214 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8210 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:55,215 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:55,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:55,219 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:55,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:55,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:55,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,223 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,224 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,224 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:55,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:55,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:55,228 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,228 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:55,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a591361[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:55,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:55,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:55,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:55,232 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:55,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:55,235 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:55,235 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:55,237 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,237 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,237 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,237 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,239 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:55,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,242 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,242 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:55,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601295306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601295312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601295317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,344 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:55,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:55,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:55,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:55,350 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:55,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:55,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:55,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,352 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:55,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:55,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:55,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:55,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:55,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:55,365 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,365 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6545d4a6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:55,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:55,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:55,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:55,390 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:55,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:55,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:55,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:55,390 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:55,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,410 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,410 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,411 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:55,411 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:55,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/83632264ddfb40ba955c9f75f8dfe209 2023-03-23 19:53:55,511 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:55,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:55,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:55,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:55,520 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/e59c11e8f8e942fd87394d4bb51f2139 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139 2023-03-23 19:53:55,523 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,523 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:55,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:55,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:55,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:55,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:55,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:55,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:55,529 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139, entries=200, sequenceid=362, filesize=14.3 K 2023-03-23 19:53:55,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/5dba76e04bb941a880c41a9bb10614a1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1 2023-03-23 19:53:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66e6234e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:55,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:55,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,536 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:55,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:55,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1, entries=150, sequenceid=362, filesize=11.9 K 2023-03-23 19:53:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/83632264ddfb40ba955c9f75f8dfe209 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209 2023-03-23 19:53:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:55,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:55,544 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:55,544 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:55,544 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:55,544 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:55,544 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:55,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:55,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:55,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:55,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209, entries=150, sequenceid=362, filesize=11.9 K 2023-03-23 19:53:55,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 916ms, sequenceid=362, compaction requested=true 2023-03-23 19:53:55,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:55,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:55,557 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:55,557 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:55,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:55,557 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:55,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:55,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:55,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:55,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:55,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:55,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:55,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,558 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:55,558 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:55,558 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:55,558 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:55,559 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,559 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,559 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec0b4d5b077e421483b539f10e4244b4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=38.7 K 2023-03-23 19:53:55,559 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6ae6be1d09204a919223b3c3a68f8e16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.3 K 2023-03-23 19:53:55,559 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ec0b4d5b077e421483b539f10e4244b4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601231468 2023-03-23 19:53:55,559 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6ae6be1d09204a919223b3c3a68f8e16, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601231468 2023-03-23 19:53:55,560 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e6a8fdfb9a3d4a1e934e0485273bc8a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679601233679 2023-03-23 19:53:55,560 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ef1ddb04a2784942811437233761461f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679601233679 2023-03-23 19:53:55,564 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e59c11e8f8e942fd87394d4bb51f2139, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234022 2023-03-23 19:53:55,564 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5dba76e04bb941a880c41a9bb10614a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234023 2023-03-23 19:53:55,575 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:55,576 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:55,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/bd8c5f9d2d1b49a6972587ec49938fea as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/bd8c5f9d2d1b49a6972587ec49938fea 2023-03-23 19:53:55,631 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into bd8c5f9d2d1b49a6972587ec49938fea(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:55,632 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:55,632 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601235557; duration=0sec 2023-03-23 19:53:55,632 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:55,632 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:53:55,633 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:53:55,633 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:55,633 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,633 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/781c15e1e6934a8a913fd60251581960, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.3 K 2023-03-23 19:53:55,633 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 781c15e1e6934a8a913fd60251581960, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679601231468 2023-03-23 19:53:55,634 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a142c37b18f44a2b83d6d42794ff9586, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679601233679 2023-03-23 19:53:55,634 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 83632264ddfb40ba955c9f75f8dfe209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234023 2023-03-23 19:53:55,644 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:55,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:55,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:55,662 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:55,662 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,662 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:55,662 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:55,662 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,663 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,664 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:55,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:55,664 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:55,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:55,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:55,697 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:55,697 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:55,697 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:55,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:55,728 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/9d97c7868ced4edbb34639dd11f4d6b4 2023-03-23 19:53:55,752 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8a9f05bda2ba414d8edab0edc3386f89 2023-03-23 19:53:55,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:55,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:55,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601295829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601295835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601295844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601295940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601295945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:55,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:55,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601295948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,006 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b6fef5744d77448aa3684b2bcfcfb3dd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b6fef5744d77448aa3684b2bcfcfb3dd 2023-03-23 19:53:56,012 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into b6fef5744d77448aa3684b2bcfcfb3dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:56,012 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:56,012 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601235557; duration=0sec 2023-03-23 19:53:56,012 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:56,057 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/4bcc2ee1346e48f29a6ef4b0c95562ec as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/4bcc2ee1346e48f29a6ef4b0c95562ec 2023-03-23 19:53:56,065 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 4bcc2ee1346e48f29a6ef4b0c95562ec(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:56,065 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:56,065 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601235557; duration=0sec 2023-03-23 19:53:56,065 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:56,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601296142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601296147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601296153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,172 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/32925b3056fd4ab1a651b84ba3a92d7a 2023-03-23 19:53:56,178 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/9d97c7868ced4edbb34639dd11f4d6b4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4 2023-03-23 19:53:56,183 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4, entries=150, sequenceid=398, filesize=11.9 K 2023-03-23 19:53:56,184 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/8a9f05bda2ba414d8edab0edc3386f89 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89 2023-03-23 19:53:56,196 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89, entries=150, sequenceid=398, filesize=11.9 K 2023-03-23 19:53:56,197 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/32925b3056fd4ab1a651b84ba3a92d7a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a 2023-03-23 19:53:56,203 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a, entries=150, sequenceid=398, filesize=11.9 K 2023-03-23 19:53:56,204 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 507ms, sequenceid=398, compaction requested=false 2023-03-23 19:53:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:56,205 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:56,207 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11302555[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:56,211 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:56,211 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:56,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:56,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:56,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:56,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:56,232 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:56,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:56,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:56,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:56,232 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:56,236 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,236 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:56,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,236 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,237 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,241 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:56,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:56,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:56,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:56,246 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:56,246 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:56,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:56,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,246 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,246 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,246 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,246 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,246 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:56,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:56,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:56,351 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:56,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:56,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:56,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:56,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,354 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:56,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:56,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:56,355 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:56,355 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:56,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:56,360 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:56,360 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:56,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,379 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/f172aae99a4049e79e6cc231eff193d3 2023-03-23 19:53:56,395 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/668dd9f4b7c14612b6fc4ecfe02b1822 2023-03-23 19:53:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:56,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:56,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601296482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601296482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601296488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601296589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601296590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601296591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601296793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601296794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:56,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601296795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,822 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/b5b5638f88ee4244954ec7f5ca422dce 2023-03-23 19:53:56,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/f172aae99a4049e79e6cc231eff193d3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3 2023-03-23 19:53:56,835 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3, entries=150, sequenceid=412, filesize=11.9 K 2023-03-23 19:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/668dd9f4b7c14612b6fc4ecfe02b1822 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822 2023-03-23 19:53:56,841 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822, entries=150, sequenceid=412, filesize=11.9 K 2023-03-23 19:53:56,844 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/b5b5638f88ee4244954ec7f5ca422dce as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce 2023-03-23 19:53:56,849 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce, entries=150, sequenceid=412, filesize=11.9 K 2023-03-23 19:53:56,851 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 491ms, sequenceid=412, compaction requested=true 2023-03-23 19:53:56,851 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:56,851 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:56,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6300fe09[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:56,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:56,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:56,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:56,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:56,862 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:56,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:56,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:56,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:56,862 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:56,864 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,864 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:56,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,864 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:56,865 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:56,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:56,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:56,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:56,869 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:56,869 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:56,869 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:56,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:56,870 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:56,870 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,969 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:56,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:56,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:56,973 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:56,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:56,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:56,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:56,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,974 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:56,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:56,975 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:56,975 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:56,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:56,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:56,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:56,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:56,977 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:56,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:56,986 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/1524b1a3697b4e0abe4aa76f14e160d0 2023-03-23 19:53:57,000 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/59e95f15cf1f4fa7bfa4256de194d3ea 2023-03-23 19:53:57,016 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9273680ccb7445f2b0b7d73757725fc4 2023-03-23 19:53:57,022 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/1524b1a3697b4e0abe4aa76f14e160d0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0 2023-03-23 19:53:57,027 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0, entries=150, sequenceid=432, filesize=11.9 K 2023-03-23 19:53:57,028 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/59e95f15cf1f4fa7bfa4256de194d3ea as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea 2023-03-23 19:53:57,034 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea, entries=150, sequenceid=432, filesize=11.9 K 2023-03-23 19:53:57,035 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9273680ccb7445f2b0b7d73757725fc4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4 2023-03-23 19:53:57,042 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4, entries=150, sequenceid=432, filesize=11.9 K 2023-03-23 19:53:57,043 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 66ms, sequenceid=432, compaction requested=true 2023-03-23 19:53:57,043 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:57,043 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282be62b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,050 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,056 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,056 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,062 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,065 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:57,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,065 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,065 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:57,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:57,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:57,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601297131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679601237122, max timestamp=1679601237131Num uniques 859; 2023-03-23 19:53:57,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601237122, max timestamp=1679601237131Num uniques 859; 2023-03-23 19:53:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679601237122, max timestamp=1679601237131Num uniques 859; 2023-03-23 19:53:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:53:57,165 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,168 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,169 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,169 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,169 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,169 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,170 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,170 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,173 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,173 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,173 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,173 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601297172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,176 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601297174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f004c33[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,180 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,189 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,189 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,204 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,204 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601297234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601297275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601297279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,304 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,307 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,312 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,312 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,312 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,312 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,316 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8ac40b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,316 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,322 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,322 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,324 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,324 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,324 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,326 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,330 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,330 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,330 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,430 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,434 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601297437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9e8f45[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,448 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,448 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,452 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,455 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,456 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601297478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601297482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/2191c61a06784573bf9e5eebaab59186 2023-03-23 19:53:57,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/e1962871bc9347c48029b6b3fd253c5e 2023-03-23 19:53:57,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,559 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,564 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,564 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,566 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,566 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,566 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,566 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf33aa3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,570 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,570 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,575 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,576 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,690 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f90da[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,703 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,706 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,706 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,713 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601297740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601297783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:57,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601297785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,817 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,824 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,824 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,824 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,824 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,827 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fca660[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,838 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,838 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,839 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,839 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,840 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,850 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:57,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,853 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:57,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:57,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9a4dfe2c68a14854af1a347e4bf9e12a 2023-03-23 19:53:57,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:57,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:57,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:57,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:57,960 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:57,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:57,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:57,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/2191c61a06784573bf9e5eebaab59186 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186 2023-03-23 19:53:57,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:57,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:57,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:57,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:57,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:57,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:57,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186, entries=200, sequenceid=443, filesize=14.3 K 2023-03-23 19:53:57,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/e1962871bc9347c48029b6b3fd253c5e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e 2023-03-23 19:53:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:57,972 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,972 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:57,972 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:57,972 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:57,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e, entries=150, sequenceid=443, filesize=11.9 K 2023-03-23 19:53:57,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9a4dfe2c68a14854af1a347e4bf9e12a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a 2023-03-23 19:53:57,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,975 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4767e29d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:57,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:57,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:57,979 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:57,979 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:57,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:57,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:57,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a, entries=150, sequenceid=443, filesize=11.9 K 2023-03-23 19:53:57,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:57,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:57,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:57,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 878ms, sequenceid=443, compaction requested=true 2023-03-23 19:53:57,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:57,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:57,982 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:57,982 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:57,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:57,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,985 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:57,985 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:53:57,985 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:57,985 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b6fef5744d77448aa3684b2bcfcfb3dd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=62.6 K 2023-03-23 19:53:57,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:57,985 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:57,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:57,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:57,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:57,986 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:57,986 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:53:57,986 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:57,986 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/bd8c5f9d2d1b49a6972587ec49938fea, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=60.2 K 2023-03-23 19:53:57,986 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b6fef5744d77448aa3684b2bcfcfb3dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234023 2023-03-23 19:53:57,987 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9d97c7868ced4edbb34639dd11f4d6b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1679601234688 2023-03-23 19:53:57,987 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bd8c5f9d2d1b49a6972587ec49938fea, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234023 2023-03-23 19:53:57,988 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f172aae99a4049e79e6cc231eff193d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1679601235827 2023-03-23 19:53:57,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:57,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,988 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 1524b1a3697b4e0abe4aa76f14e160d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679601236468 2023-03-23 19:53:57,989 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8a9f05bda2ba414d8edab0edc3386f89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1679601234688 2023-03-23 19:53:57,989 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:57,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:57,989 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 668dd9f4b7c14612b6fc4ecfe02b1822, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1679601235827 2023-03-23 19:53:57,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:57,990 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 59e95f15cf1f4fa7bfa4256de194d3ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679601236468 2023-03-23 19:53:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:57,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,991 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 2191c61a06784573bf9e5eebaab59186, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:53:57,991 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e1962871bc9347c48029b6b3fd253c5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:53:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:57,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:58,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:58,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:58,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:58,011 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:58,011 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:58,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:58,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:58,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:58,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:58,021 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#81 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:58,026 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:58,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:58,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:58,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:58,115 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:58,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:58,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:58,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:58,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:58,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:58,117 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:58,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:58,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:58,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:58,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:58,118 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:58,119 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:53:58,128 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b3a128e67cf94a61a2cb51e411cc7194 2023-03-23 19:53:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:58,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:58,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601298287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601298291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601298293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601298392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,445 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/648833f479ec446b9289a1cf1206b535 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/648833f479ec446b9289a1cf1206b535 2023-03-23 19:53:58,458 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/fb0c8d0d942f41eaad906e0c74d0d41c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/fb0c8d0d942f41eaad906e0c74d0d41c 2023-03-23 19:53:58,466 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into 648833f479ec446b9289a1cf1206b535(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:58,466 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:58,466 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=11, startTime=1679601237982; duration=0sec 2023-03-23 19:53:58,466 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:53:58,466 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:53:58,469 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:53:58,469 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:53:58,469 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into fb0c8d0d942f41eaad906e0c74d0d41c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:58,469 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:58,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:58,469 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=11, startTime=1679601237982; duration=0sec 2023-03-23 19:53:58,469 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/4bcc2ee1346e48f29a6ef4b0c95562ec, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=60.2 K 2023-03-23 19:53:58,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:58,470 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4bcc2ee1346e48f29a6ef4b0c95562ec, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601234023 2023-03-23 19:53:58,470 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 32925b3056fd4ab1a651b84ba3a92d7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1679601234688 2023-03-23 19:53:58,471 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b5b5638f88ee4244954ec7f5ca422dce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1679601235827 2023-03-23 19:53:58,471 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9273680ccb7445f2b0b7d73757725fc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679601236468 2023-03-23 19:53:58,472 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9a4dfe2c68a14854af1a347e4bf9e12a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:53:58,483 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#85 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:53:58,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47888 deadline: 1679601298516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,521 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:53:58,548 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/81731046a1ba41e68cc872e373ee87b5 2023-03-23 19:53:58,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601298597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,899 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9c6a4cbeb58746d385a26a1100b72041 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9c6a4cbeb58746d385a26a1100b72041 2023-03-23 19:53:58,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:58,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601298901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:58,905 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 9c6a4cbeb58746d385a26a1100b72041(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:53:58,905 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:58,905 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=11, startTime=1679601237982; duration=0sec 2023-03-23 19:53:58,905 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:53:58,972 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/93d4ee317a144a60912c3d0f56d56d11 2023-03-23 19:53:58,979 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b3a128e67cf94a61a2cb51e411cc7194 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194 2023-03-23 19:53:58,985 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194, entries=150, sequenceid=478, filesize=11.9 K 2023-03-23 19:53:58,986 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/81731046a1ba41e68cc872e373ee87b5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5 2023-03-23 19:53:58,993 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5, entries=150, sequenceid=478, filesize=11.9 K 2023-03-23 19:53:58,994 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/93d4ee317a144a60912c3d0f56d56d11 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11 2023-03-23 19:53:58,998 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11, entries=150, sequenceid=478, filesize=11.9 K 2023-03-23 19:53:58,999 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 880ms, sequenceid=478, compaction requested=false 2023-03-23 19:53:58,999 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:53:58,999 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:59,060 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:59,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2224eef0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:59,062 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:59,062 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,122 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:59,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:59,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:59,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:59,187 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:59,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:59,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:59,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:59,187 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:59,229 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,229 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,229 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:59,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:59,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:59,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:59,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:59,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:59,292 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:59,292 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:59,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:59,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:53:59,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:53:59,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:53:59,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:59,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:53:59,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:59,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:53:59,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:53:59,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601299324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601299327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,392 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:59,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:59,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:59,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:59,395 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:59,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:59,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:59,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:59,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601299404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,416 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:59,416 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:59,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:59,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:59,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:59,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:59,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:59,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:59,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:59,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601299428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601299428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:59,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb0432e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:59,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,499 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:59,500 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,500 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:59,537 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:59,541 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:59,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:59,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:59,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:59,541 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:59,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:59,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,570 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,630 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:59,630 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:59,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:53:59,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:53:59,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:53:59,630 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:59,630 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:59,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:53:59,630 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:53:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:59,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601299631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601299631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/eb4cc369556b4d2cb30f25b22af53aa6 2023-03-23 19:53:59,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:53:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:53:59,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:53:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:53:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:53:59,735 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:53:59,735 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:59,735 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:53:59,735 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:53:59,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:53:59,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:53:59,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:53:59,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:53:59,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:53:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:53:59,762 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:59,762 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:53:59,762 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:53:59,762 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:53:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:53:59,799 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:53:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:53:59,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:53:59,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2211e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:53:59,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:53:59,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:53:59,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:53:59,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:53:59,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:53:59,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:53:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:53:59,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:53:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:53:59,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:53:59,887 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:53:59,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:53:59,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:53:59,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:53:59,888 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:53:59,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601299934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:53:59,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601299934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:53:59,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:53:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:53:59,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:53:59,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:00,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:00,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:00,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:00,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,015 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:00,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,015 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:00,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:00,115 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:00,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:00,118 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:00,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:00,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:00,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6f91d99700fa4d238d1c28bda1be85b6 2023-03-23 19:54:00,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:00,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:00,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:00,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:00,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:00,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:00,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:00,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:00,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:00,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aa948dd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:00,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:00,278 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:00,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:00,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:00,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:00,340 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:00,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:00,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:00,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:00,340 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:00,391 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,391 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,391 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:00,392 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:00,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601300410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:00,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:00,429 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:00,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:00,430 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:00,430 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:00,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:00,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601300438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601300441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,530 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:00,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:00,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:00,534 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:00,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:00,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:00,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,534 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,545 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,546 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,546 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:00,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:00,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:00,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:00,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fff6a16[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:00,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:00,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/8f7f2c0a08a1498c83807b6c1a30061c 2023-03-23 19:54:00,560 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:00,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:00,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/eb4cc369556b4d2cb30f25b22af53aa6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6 2023-03-23 19:54:00,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,567 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6, entries=200, sequenceid=493, filesize=14.3 K 2023-03-23 19:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6f91d99700fa4d238d1c28bda1be85b6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6 2023-03-23 19:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:00,572 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:00,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:00,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:00,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:00,572 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:00,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6, entries=150, sequenceid=493, filesize=11.9 K 2023-03-23 19:54:00,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/8f7f2c0a08a1498c83807b6c1a30061c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c 2023-03-23 19:54:00,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:00,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:00,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:00,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:00,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c, entries=150, sequenceid=493, filesize=11.9 K 2023-03-23 19:54:00,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 1287ms, sequenceid=493, compaction requested=true 2023-03-23 19:54:00,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:00,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:00,582 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:00,582 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:00,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:00,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:00,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:00,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:00,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:00,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:00,584 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:00,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:00,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:00,584 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:00,584 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:54:00,584 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,585 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/648833f479ec446b9289a1cf1206b535, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=39.0 K 2023-03-23 19:54:00,585 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:00,585 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 648833f479ec446b9289a1cf1206b535, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:54:00,585 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:54:00,585 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,585 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/fb0c8d0d942f41eaad906e0c74d0d41c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.6 K 2023-03-23 19:54:00,586 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b3a128e67cf94a61a2cb51e411cc7194, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679601237160 2023-03-23 19:54:00,586 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting fb0c8d0d942f41eaad906e0c74d0d41c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:54:00,586 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting eb4cc369556b4d2cb30f25b22af53aa6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:00,586 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 81731046a1ba41e68cc872e373ee87b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679601237160 2023-03-23 19:54:00,587 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6f91d99700fa4d238d1c28bda1be85b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:00,594 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:00,600 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:00,616 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6a1dcb21e2d54add849d3544b40381d6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6a1dcb21e2d54add849d3544b40381d6 2023-03-23 19:54:00,622 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into 6a1dcb21e2d54add849d3544b40381d6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:00,622 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:00,622 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601240582; duration=0sec 2023-03-23 19:54:00,623 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:00,623 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:00,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:00,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:54:00,624 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,624 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9c6a4cbeb58746d385a26a1100b72041, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.6 K 2023-03-23 19:54:00,625 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9c6a4cbeb58746d385a26a1100b72041, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1679601237100 2023-03-23 19:54:00,625 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 93d4ee317a144a60912c3d0f56d56d11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679601237160 2023-03-23 19:54:00,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8f7f2c0a08a1498c83807b6c1a30061c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:00,632 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:00,648 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/7f15f4d75ffe48c9b2a495cc3af1a253 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7f15f4d75ffe48c9b2a495cc3af1a253 2023-03-23 19:54:00,657 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 7f15f4d75ffe48c9b2a495cc3af1a253(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:00,658 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:00,658 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601240583; duration=0sec 2023-03-23 19:54:00,658 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:00,684 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:00,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:00,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:00,689 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:00,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:00,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:00,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:00,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:00,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:00,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:00,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:00,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:00,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:00,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:00,693 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:00,693 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:00,693 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:54:00,693 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:00,693 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:00,694 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:00,694 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:00,694 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:00,694 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:01,013 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/3f63204356d647d993f52d9134c9313a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/3f63204356d647d993f52d9134c9313a 2023-03-23 19:54:01,022 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into 3f63204356d647d993f52d9134c9313a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:01,022 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,022 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601240582; duration=0sec 2023-03-23 19:54:01,022 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:01,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ec6dc2f817c544aea4853f6caf8e28fc 2023-03-23 19:54:01,131 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb 2023-03-23 19:54:01,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/f9a3da868417499aa63a1906a0ff49ee 2023-03-23 19:54:01,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ec6dc2f817c544aea4853f6caf8e28fc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc 2023-03-23 19:54:01,155 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc, entries=150, sequenceid=515, filesize=11.9 K 2023-03-23 19:54:01,156 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb 2023-03-23 19:54:01,161 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb, entries=150, sequenceid=515, filesize=11.9 K 2023-03-23 19:54:01,162 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/f9a3da868417499aa63a1906a0ff49ee as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee 2023-03-23 19:54:01,167 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee, entries=150, sequenceid=515, filesize=11.9 K 2023-03-23 19:54:01,168 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 475ms, sequenceid=515, compaction requested=false 2023-03-23 19:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,168 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,171 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228beff4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,175 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,176 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,179 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,179 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,186 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:01,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,186 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,287 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,296 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,298 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31b2906a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,325 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,325 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,430 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,432 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:01,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,437 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1edc921[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,439 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,439 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,441 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,447 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,447 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:01,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:01,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,456 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,470 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/844e41085f234b7ebda1458d58f376bb 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241481, max timestamp=1679601241490Num uniques 859; 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241481, max timestamp=1679601241490Num uniques 859; 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241481, max timestamp=1679601241490Num uniques 859; 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73 2023-03-23 19:54:01,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:01,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601301548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:01,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601301548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,581 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,584 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,584 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:01,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:01,592 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/b8d66ab927af47fc8ae2a69ee10421ab 2023-03-23 19:54:01,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@546bcd0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/844e41085f234b7ebda1458d58f376bb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb 2023-03-23 19:54:01,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb, entries=250, sequenceid=529, filesize=16.7 K 2023-03-23 19:54:01,608 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-23 19:54:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,608 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,608 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,608 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73 2023-03-23 19:54:01,610 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,610 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,610 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,611 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73, entries=150, sequenceid=529, filesize=11.9 K 2023-03-23 19:54:01,616 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:01,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,617 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,617 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/b8d66ab927af47fc8ae2a69ee10421ab as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab 2023-03-23 19:54:01,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab, entries=150, sequenceid=529, filesize=11.9 K 2023-03-23 19:54:01,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 171ms, sequenceid=529, compaction requested=true 2023-03-23 19:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:01,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:01,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:01,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:01,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42460 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:01,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:54:01,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:01,626 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:54:01,626 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/3f63204356d647d993f52d9134c9313a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=41.5 K 2023-03-23 19:54:01,626 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,626 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6a1dcb21e2d54add849d3544b40381d6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.7 K 2023-03-23 19:54:01,626 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 3f63204356d647d993f52d9134c9313a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:01,627 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6a1dcb21e2d54add849d3544b40381d6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:01,627 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ec6dc2f817c544aea4853f6caf8e28fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679601239319 2023-03-23 19:54:01,627 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 50cf8aecc92f45f5bb6e4a5dcbdb1abb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679601239319 2023-03-23 19:54:01,628 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 844e41085f234b7ebda1458d58f376bb, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:01,628 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5c56dc2a7ac4486b9bb79e6cd8ef8e73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:01,639 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#99 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:01,641 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:01,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:01,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241537, max timestamp=1679601241656Num uniques 1018; 2023-03-23 19:54:01,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241537, max timestamp=1679601241656Num uniques 1018; 2023-03-23 19:54:01,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241537, max timestamp=1679601241656Num uniques 1018; 2023-03-23 19:54:01,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=33.54 KB heapSize=80.63 KB 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:01,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:01,676 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ea44e72997754d1a82de479fd9d9344e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea44e72997754d1a82de479fd9d9344e 2023-03-23 19:54:01,682 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into ea44e72997754d1a82de479fd9d9344e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:01,682 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:01,682 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601241624; duration=0sec 2023-03-23 19:54:01,682 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:01,682 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:01,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ea57b74aecf443589d9d884c7f6b9e56 2023-03-23 19:54:01,684 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:01,684 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:54:01,684 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,684 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7f15f4d75ffe48c9b2a495cc3af1a253, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.7 K 2023-03-23 19:54:01,686 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7f15f4d75ffe48c9b2a495cc3af1a253, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679601238260 2023-03-23 19:54:01,687 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f9a3da868417499aa63a1906a0ff49ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679601239319 2023-03-23 19:54:01,687 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b8d66ab927af47fc8ae2a69ee10421ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:01,696 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:01,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241694, max timestamp=1679601241699Num uniques 859; 2023-03-23 19:54:01,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241694, max timestamp=1679601241699Num uniques 859; 2023-03-23 19:54:01,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601241694, max timestamp=1679601241699Num uniques 859; 2023-03-23 19:54:01,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:01,717 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,721 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,723 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:01,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,732 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,732 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,732 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:01,732 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,735 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78cf07c3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,738 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,746 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,746 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,752 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,753 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241716, max timestamp=1679601241732Num uniques 1018; 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241716, max timestamp=1679601241732Num uniques 1018; 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601241716, max timestamp=1679601241732Num uniques 1018; 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679601241779, max timestamp=1679601241789Num uniques 1077; 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679601241779, max timestamp=1679601241789Num uniques 1077; 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679601241779, max timestamp=1679601241789Num uniques 1077; 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-23 19:54:01,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-23 19:54:01,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601301821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679601241807, max timestamp=1679601241821Num uniques 1186; 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679601241807, max timestamp=1679601241821Num uniques 1186; 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679601241807, max timestamp=1679601241821Num uniques 1186; 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-03-23 19:54:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601301822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-03-23 19:54:01,853 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,856 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:01,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,860 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,860 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,860 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:01,860 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,862 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddbf970[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,867 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,868 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,869 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,869 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:01,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:01,871 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:01,871 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:01,871 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:01,871 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:01,871 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:01,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:01,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:01,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,880 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:01,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,880 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:01,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:01,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601241938, max timestamp=1679601241950Num uniques 1195; 2023-03-23 19:54:01,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601241938, max timestamp=1679601241950Num uniques 1195; 2023-03-23 19:54:01,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-03-23 19:54:01,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-03-23 19:54:01,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601241938, max timestamp=1679601241950Num uniques 1195; 2023-03-23 19:54:01,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-03-23 19:54:01,980 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:01,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:01,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:01,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:01,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:01,985 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:01,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:01,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:01,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679601241973, max timestamp=1679601241983Num uniques 1104; 2023-03-23 19:54:01,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679601241973, max timestamp=1679601241983Num uniques 1104; 2023-03-23 19:54:01,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-23 19:54:01,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-23 19:54:01,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679601241973, max timestamp=1679601241983Num uniques 1104; 2023-03-23 19:54:01,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-23 19:54:01,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,987 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:01,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:01,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:01,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:01,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:01,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:01,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:01,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:01,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:01,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:01,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:01,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3210972b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:01,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,994 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:01,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:01,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:01,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:01,995 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,995 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:01,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,996 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:02,000 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:02,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:02,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:02,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:02,000 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:02,004 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,004 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,004 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,005 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:02,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:02,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:02,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679601242003, max timestamp=1679601242022Num uniques 1113; 2023-03-23 19:54:02,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-03-23 19:54:02,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679601242003, max timestamp=1679601242022Num uniques 1113; 2023-03-23 19:54:02,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-03-23 19:54:02,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1679601242003, max timestamp=1679601242022Num uniques 1113; 2023-03-23 19:54:02,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-03-23 19:54:02,064 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/d527063a2ec0479cac26412c569fea84 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d527063a2ec0479cac26412c569fea84 2023-03-23 19:54:02,072 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into d527063a2ec0479cac26412c569fea84(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:02,072 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,072 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601241624; duration=0sec 2023-03-23 19:54:02,073 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679601242059, max timestamp=1679601242081Num uniques 1172; 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679601242059, max timestamp=1679601242081Num uniques 1172; 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679601242059, max timestamp=1679601242081Num uniques 1172; 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-03-23 19:54:02,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-03-23 19:54:02,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/7a0059dd78f24b91b6c7acffacaab943 2023-03-23 19:54:02,108 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/503375dec5be42df999dfb857cf5ee49 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/503375dec5be42df999dfb857cf5ee49 2023-03-23 19:54:02,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:02,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:02,114 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 503375dec5be42df999dfb857cf5ee49(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:02,114 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,114 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601241624; duration=0sec 2023-03-23 19:54:02,115 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:02,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:02,116 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:02,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:02,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:02,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d5f5f08c5e054dc8979806a5a1c24d70 2023-03-23 19:54:02,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,118 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:02,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:02,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:02,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:02,120 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:02,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:02,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:02,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/ea57b74aecf443589d9d884c7f6b9e56 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56 2023-03-23 19:54:02,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2800d85[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:02,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:02,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56, entries=250, sequenceid=567, filesize=16.7 K 2023-03-23 19:54:02,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/7a0059dd78f24b91b6c7acffacaab943 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943 2023-03-23 19:54:02,128 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:02,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:02,130 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601242111, max timestamp=1679601242133Num uniques 1081; 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601242111, max timestamp=1679601242133Num uniques 1081; 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601242111, max timestamp=1679601242133Num uniques 1081; 2023-03-23 19:54:02,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-23 19:54:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943, entries=150, sequenceid=567, filesize=11.9 K 2023-03-23 19:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/d5f5f08c5e054dc8979806a5a1c24d70 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70 2023-03-23 19:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:02,139 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:02,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:02,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:02,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:02,139 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:02,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70, entries=150, sequenceid=567, filesize=11.9 K 2023-03-23 19:54:02,142 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=33.54 KB/34350 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 483ms, sequenceid=567, compaction requested=false 2023-03-23 19:54:02,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:02,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:02,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,153 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:02,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,154 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:02,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:02,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:02,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=60.38 KB heapSize=159.09 KB 2023-03-23 19:54:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:02,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/791d81a2209d4328afee2562542fe746 2023-03-23 19:54:02,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/59c4dc7c06d5488cb29c2068bf4e355b 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601242179, max timestamp=1679601242191Num uniques 859; 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601242179, max timestamp=1679601242191Num uniques 859; 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601242179, max timestamp=1679601242191Num uniques 859; 2023-03-23 19:54:02,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:02,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601302218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:02,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:02,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:02,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:02,258 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,259 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:02,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:02,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:02,261 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,261 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:02,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:02,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:02,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@271e309[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:02,266 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:02,266 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:02,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:02,268 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,268 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,270 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:02,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:02,273 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:02,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:02,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:02,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:02,273 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:02,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,281 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,283 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,283 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:02,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,283 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,284 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:02,284 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:02,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601302321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,384 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:02,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:02,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:02,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:02,387 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:02,387 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,387 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:02,387 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:02,387 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,387 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:02,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:02,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:02,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:02,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:02,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:02,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:02,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:02,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:02,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:02,393 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:02,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e38a4e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:02,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:02,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:02,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:02,398 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:02,400 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:02,400 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:02,400 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:02,400 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:02,400 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:02,402 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,402 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,402 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,402 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,406 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,406 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:02,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:02,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,406 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,406 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:02,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:02,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:02,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47804 deadline: 1679601302426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,429 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:02,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:02,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:02,510 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:02,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:02,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:02,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,512 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:02,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:02,514 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,514 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:02,514 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:02,514 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:02,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:02,516 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:02,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36fcfcb6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:02,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:02,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,525 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:02,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601302526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:02,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:02,530 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:02,530 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:02,530 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:02,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:02,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,533 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:02,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:02,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:02,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,541 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:02,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:02,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:02,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/77818b35274f4b83b7a98fc55679aa5e 2023-03-23 19:54:02,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/791d81a2209d4328afee2562542fe746 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746 2023-03-23 19:54:02,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746, entries=150, sequenceid=732, filesize=11.9 K 2023-03-23 19:54:02,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/59c4dc7c06d5488cb29c2068bf4e355b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b 2023-03-23 19:54:02,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b, entries=150, sequenceid=732, filesize=11.9 K 2023-03-23 19:54:02,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/77818b35274f4b83b7a98fc55679aa5e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e 2023-03-23 19:54:02,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e, entries=150, sequenceid=732, filesize=11.9 K 2023-03-23 19:54:02,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~159.05 KB/162864, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 464ms, sequenceid=732, compaction requested=true 2023-03-23 19:54:02,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:02,622 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:02,622 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:02,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:02,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:02,623 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:02,623 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:54:02,623 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,624 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d527063a2ec0479cac26412c569fea84, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.8 K 2023-03-23 19:54:02,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42562 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:02,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:54:02,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d527063a2ec0479cac26412c569fea84, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:02,624 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,624 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea44e72997754d1a82de479fd9d9344e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=41.6 K 2023-03-23 19:54:02,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7a0059dd78f24b91b6c7acffacaab943, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679601241542 2023-03-23 19:54:02,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ea44e72997754d1a82de479fd9d9344e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:02,625 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 59c4dc7c06d5488cb29c2068bf4e355b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:02,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ea57b74aecf443589d9d884c7f6b9e56, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679601241537 2023-03-23 19:54:02,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 791d81a2209d4328afee2562542fe746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:02,631 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#108 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:02,633 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#109 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:02,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:02,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:02,644 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b1ac8e768fb946f4a7141fe9d7d58b16 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b1ac8e768fb946f4a7141fe9d7d58b16 2023-03-23 19:54:02,644 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/d961ca22898049b38f88f53053af1f2f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d961ca22898049b38f88f53053af1f2f 2023-03-23 19:54:02,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:02,651 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into d961ca22898049b38f88f53053af1f2f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:02,651 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,651 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=13, startTime=1679601242622; duration=0sec 2023-03-23 19:54:02,651 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:02,652 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:02,654 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into b1ac8e768fb946f4a7141fe9d7d58b16(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:02,654 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,654 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=13, startTime=1679601242622; duration=0sec 2023-03-23 19:54:02,654 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:02,654 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:02,654 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:54:02,654 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,654 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/503375dec5be42df999dfb857cf5ee49, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=36.8 K 2023-03-23 19:54:02,655 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 503375dec5be42df999dfb857cf5ee49, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679601241448 2023-03-23 19:54:02,655 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d5f5f08c5e054dc8979806a5a1c24d70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679601241542 2023-03-23 19:54:02,656 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 77818b35274f4b83b7a98fc55679aa5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:02,656 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:02,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:02,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:02,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:02,660 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:02,661 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,661 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:02,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:02,664 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#110 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:02,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:02,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:02,666 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:02,666 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:02,666 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:02,667 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:02,667 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:02,667 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:02,667 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:02,668 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:02,682 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/db98471ae6cc45dca83535d9dd605583 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/db98471ae6cc45dca83535d9dd605583 2023-03-23 19:54:02,688 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into db98471ae6cc45dca83535d9dd605583(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:02,688 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:02,688 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=13, startTime=1679601242622; duration=0sec 2023-03-23 19:54:02,688 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:02,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:02,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:02,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601302844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601302945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:02,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:02,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601302946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,096 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fdd138091114430cbc067095ea628294 2023-03-23 19:54:03,110 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/a1e21048ec74473a81f8fe654da33908 2023-03-23 19:54:03,121 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/eb6af98b590d4ae0a95076f4a8f5ebff 2023-03-23 19:54:03,126 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/fdd138091114430cbc067095ea628294 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294 2023-03-23 19:54:03,131 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294, entries=150, sequenceid=769, filesize=11.9 K 2023-03-23 19:54:03,132 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/a1e21048ec74473a81f8fe654da33908 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908 2023-03-23 19:54:03,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908, entries=150, sequenceid=769, filesize=11.9 K 2023-03-23 19:54:03,148 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/eb6af98b590d4ae0a95076f4a8f5ebff as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff 2023-03-23 19:54:03,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601303148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601303148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,157 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff, entries=150, sequenceid=769, filesize=11.9 K 2023-03-23 19:54:03,157 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 489ms, sequenceid=769, compaction requested=false 2023-03-23 19:54:03,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:03,158 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341233c9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:03,165 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:03,165 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,167 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:03,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:03,169 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:03,174 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:03,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:03,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:03,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:03,174 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:03,177 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:03,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:03,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:03,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:03,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:03,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:03,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:03,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,188 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:03,188 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:03,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:03,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:03,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:03,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:03,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:03,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:03,288 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:03,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:03,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:03,291 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:03,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:03,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:03,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:03,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,291 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:03,293 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:03,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:03,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:03,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:03,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:03,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:03,295 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. started... 2023-03-23 19:54:03,296 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:03,296 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:03,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:03,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. as already flushing 2023-03-23 19:54:03,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601303475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601243470, max timestamp=1679601243474Num uniques 859; 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601243470, max timestamp=1679601243474Num uniques 859; 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601243470, max timestamp=1679601243474Num uniques 859; 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:03,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:03,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601303507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601303577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,579 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:60301 2023-03-23 19:54:03,580 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:03,580 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:60301 2023-03-23 19:54:03,580 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:03,580 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bd6fc59 to 127.0.0.1:60301 2023-03-23 19:54:03,580 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:03,590 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:60301 2023-03-23 19:54:03,590 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:03,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601303612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,706 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b77371be78754e2fbc9113e7f28bafee 2023-03-23 19:54:03,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601303781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:03,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:03,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601303813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:04,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47830 deadline: 1679601304083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:04,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47872 deadline: 1679601304115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,125 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6107c1d535c9439bb0f92a353e8973a8 2023-03-23 19:54:04,545 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/928b6e3dd40940f6a8b0dad48f96eb2c 2023-03-23 19:54:04,551 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/b77371be78754e2fbc9113e7f28bafee as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee 2023-03-23 19:54:04,556 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee, entries=100, sequenceid=781, filesize=9.5 K 2023-03-23 19:54:04,557 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/6107c1d535c9439bb0f92a353e8973a8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8 2023-03-23 19:54:04,561 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8, entries=100, sequenceid=781, filesize=9.5 K 2023-03-23 19:54:04,562 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/928b6e3dd40940f6a8b0dad48f96eb2c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c 2023-03-23 19:54:04,567 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c, entries=100, sequenceid=781, filesize=9.5 K 2023-03-23 19:54:04,568 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 1272ms, sequenceid=781, compaction requested=true 2023-03-23 19:54:04,568 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:04,568 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:04,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:04,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:04,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:04,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:04,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f6f7e9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:04,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:04,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,574 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:04,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:04,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:04,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:04,579 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:04,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:04,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:04,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:04,580 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:04,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:04,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:04,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:04,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:04,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-23 19:54:04,591 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:60301 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:04,591 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:04,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:04,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:04,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:04,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:04,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:04,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:04,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:04,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:04,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:04,593 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:04,593 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:04,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:04,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/862eb1075bf04356a08c523a09ce1bde 2023-03-23 19:54:04,621 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:60301 2023-03-23 19:54:04,621 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:04,693 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:04,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:05,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/f4d4f40a04b844b7ae8691c27ba0f846 2023-03-23 19:54:05,281 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:60301 2023-03-23 19:54:05,281 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:05,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/7811acd08072463b8695470c7129b6ef 2023-03-23 19:54:05,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/862eb1075bf04356a08c523a09ce1bde as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde 2023-03-23 19:54:05,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde, entries=150, sequenceid=817, filesize=11.9 K 2023-03-23 19:54:05,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/f4d4f40a04b844b7ae8691c27ba0f846 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846 2023-03-23 19:54:05,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846, entries=150, sequenceid=817, filesize=11.9 K 2023-03-23 19:54:05,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/7811acd08072463b8695470c7129b6ef as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef 2023-03-23 19:54:05,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef, entries=150, sequenceid=817, filesize=11.9 K 2023-03-23 19:54:05,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=13.42 KB/13740 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 861ms, sequenceid=817, compaction requested=true 2023-03-23 19:54:05,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:05,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:05,452 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:05,452 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:05,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:05,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:05,453 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:05,453 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/A is initiating minor compaction (all files) 2023-03-23 19:54:05,453 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/A in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:05,454 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b1ac8e768fb946f4a7141fe9d7d58b16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=46.4 K 2023-03-23 19:54:05,454 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:05,454 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/B is initiating minor compaction (all files) 2023-03-23 19:54:05,454 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/B in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:05,454 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d961ca22898049b38f88f53053af1f2f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=46.4 K 2023-03-23 19:54:05,454 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b1ac8e768fb946f4a7141fe9d7d58b16, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:05,454 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d961ca22898049b38f88f53053af1f2f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:05,454 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fdd138091114430cbc067095ea628294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1679601242211 2023-03-23 19:54:05,455 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a1e21048ec74473a81f8fe654da33908, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1679601242211 2023-03-23 19:54:05,455 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b77371be78754e2fbc9113e7f28bafee, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679601242836 2023-03-23 19:54:05,455 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6107c1d535c9439bb0f92a353e8973a8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679601242836 2023-03-23 19:54:05,455 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 862eb1075bf04356a08c523a09ce1bde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679601243504 2023-03-23 19:54:05,455 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f4d4f40a04b844b7ae8691c27ba0f846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679601243504 2023-03-23 19:54:05,467 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#B#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:05,468 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#A#compaction#121 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:05,884 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/d0ae5cf005e740a1b6d57a5c6dd9a735 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/d0ae5cf005e740a1b6d57a5c6dd9a735 2023-03-23 19:54:05,885 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/c193cac4955545cdb9ac70c99ee13cf8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/c193cac4955545cdb9ac70c99ee13cf8 2023-03-23 19:54:05,889 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/A of 01acdc2ff8c96b2160b74ed8a9a21f8e into d0ae5cf005e740a1b6d57a5c6dd9a735(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:05,889 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:05,889 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/A, priority=12, startTime=1679601245452; duration=0sec 2023-03-23 19:54:05,889 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:05,890 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:05,890 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/B of 01acdc2ff8c96b2160b74ed8a9a21f8e into c193cac4955545cdb9ac70c99ee13cf8(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:05,890 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:05,890 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/B, priority=12, startTime=1679601245452; duration=0sec 2023-03-23 19:54:05,890 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:05,891 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:05,891 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 01acdc2ff8c96b2160b74ed8a9a21f8e/C is initiating minor compaction (all files) 2023-03-23 19:54:05,891 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 01acdc2ff8c96b2160b74ed8a9a21f8e/C in TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:05,891 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/db98471ae6cc45dca83535d9dd605583, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp, totalSize=46.4 K 2023-03-23 19:54:05,891 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting db98471ae6cc45dca83535d9dd605583, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1679601242133 2023-03-23 19:54:05,892 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting eb6af98b590d4ae0a95076f4a8f5ebff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1679601242211 2023-03-23 19:54:05,892 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 928b6e3dd40940f6a8b0dad48f96eb2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679601242836 2023-03-23 19:54:05,892 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7811acd08072463b8695470c7129b6ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679601243504 2023-03-23 19:54:05,899 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 01acdc2ff8c96b2160b74ed8a9a21f8e#C#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:06,309 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9743c2e3b25a486eb71bc16f2bc8bc4e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9743c2e3b25a486eb71bc16f2bc8bc4e 2023-03-23 19:54:06,314 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 01acdc2ff8c96b2160b74ed8a9a21f8e/C of 01acdc2ff8c96b2160b74ed8a9a21f8e into 9743c2e3b25a486eb71bc16f2bc8bc4e(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:06,314 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:06,314 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e., storeName=01acdc2ff8c96b2160b74ed8a9a21f8e/C, priority=12, startTime=1679601245452; duration=0sec 2023-03-23 19:54:06,314 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:06,458 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:60301 2023-03-23 19:54:06,458 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:08,565 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:60301 2023-03-23 19:54:08,565 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:08,565 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 264 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 251 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4804 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4949 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2098 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6292 rows 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2109 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6320 rows 2023-03-23 19:54:08,566 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:54:08,566 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b17e1c1 to 127.0.0.1:60301 2023-03-23 19:54:08,566 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:08,571 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:54:08,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:54:08,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:08,585 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601248585"}]},"ts":"1679601248585"} 2023-03-23 19:54:08,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:08,587 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:54:08,588 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:54:08,590 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, UNASSIGN}] 2023-03-23 19:54:08,591 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, UNASSIGN 2023-03-23 19:54:08,592 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=01acdc2ff8c96b2160b74ed8a9a21f8e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:08,592 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601248592"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601248592"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601248592"}]},"ts":"1679601248592"} 2023-03-23 19:54:08,594 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:08,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 01acdc2ff8c96b2160b74ed8a9a21f8e, disabling compactions & flushes 2023-03-23 19:54:08,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. after waiting 0 ms 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:08,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 01acdc2ff8c96b2160b74ed8a9a21f8e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=A 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:08,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=B 2023-03-23 19:54:08,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:08,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 01acdc2ff8c96b2160b74ed8a9a21f8e, store=C 2023-03-23 19:54:08,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:08,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:09,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/31bdc8a85c584251ada3c429c2a130c0 2023-03-23 19:54:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:09,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9cb43f80269442dd9d1afa2e2489cbd6 2023-03-23 19:54:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:09,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9472703abf3c4f04918ff2928ef72074 2023-03-23 19:54:09,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/A/31bdc8a85c584251ada3c429c2a130c0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/31bdc8a85c584251ada3c429c2a130c0 2023-03-23 19:54:09,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/31bdc8a85c584251ada3c429c2a130c0, entries=50, sequenceid=827, filesize=7.1 K 2023-03-23 19:54:09,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/B/9cb43f80269442dd9d1afa2e2489cbd6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9cb43f80269442dd9d1afa2e2489cbd6 2023-03-23 19:54:09,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9cb43f80269442dd9d1afa2e2489cbd6, entries=50, sequenceid=827, filesize=7.1 K 2023-03-23 19:54:09,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/.tmp/C/9472703abf3c4f04918ff2928ef72074 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9472703abf3c4f04918ff2928ef72074 2023-03-23 19:54:10,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9472703abf3c4f04918ff2928ef72074, entries=50, sequenceid=827, filesize=7.1 K 2023-03-23 19:54:10,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 01acdc2ff8c96b2160b74ed8a9a21f8e in 1251ms, sequenceid=827, compaction requested=false 2023-03-23 19:54:10,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f5c214a4732a4beaa5a28ea532f2995f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/be3d252b16a54e8bac2d46ac1232c1f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b48dd99ef3064b08b29cd49338d31faa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/bd6ccc5856454524a440b77047cb1709, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec0b4d5b077e421483b539f10e4244b4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b6fef5744d77448aa3684b2bcfcfb3dd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/648833f479ec446b9289a1cf1206b535, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/3f63204356d647d993f52d9134c9313a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea44e72997754d1a82de479fd9d9344e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b1ac8e768fb946f4a7141fe9d7d58b16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde] to archive 2023-03-23 19:54:10,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:10,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fb58c109a4824554bc2666f2a2b19a96 2023-03-23 19:54:10,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/597e0d2d969e4c88a68ee5ce68427063 2023-03-23 19:54:10,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f5c214a4732a4beaa5a28ea532f2995f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f5c214a4732a4beaa5a28ea532f2995f 2023-03-23 19:54:10,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/5e14b7cfb6004248bbc4fa3641847c69 2023-03-23 19:54:10,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fd47757b61fc44549941ef6630c0411c 2023-03-23 19:54:10,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/53276d16997f44f0842dfb0a0878fae5 2023-03-23 19:54:10,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/55474feaa4b6454ab4133caf04cee0d6 2023-03-23 19:54:10,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/be3d252b16a54e8bac2d46ac1232c1f1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/be3d252b16a54e8bac2d46ac1232c1f1 2023-03-23 19:54:10,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e2b15a2ba6f94cd3a9c045b9551ec3a0 2023-03-23 19:54:10,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/20f46d5b285e4efcaf3916c42d497719 2023-03-23 19:54:10,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/880977e8338644bf9e485298e2cb43ed 2023-03-23 19:54:10,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b48dd99ef3064b08b29cd49338d31faa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b48dd99ef3064b08b29cd49338d31faa 2023-03-23 19:54:10,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/324522bd8f564510ad269aa30a17fd12 2023-03-23 19:54:10,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/11437c9b93d3464691ca2cb4b10e8c86 2023-03-23 19:54:10,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/bd6ccc5856454524a440b77047cb1709 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/bd6ccc5856454524a440b77047cb1709 2023-03-23 19:54:10,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/03d027075f2c4d4983ac35452a0c88c4 2023-03-23 19:54:10,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/18f9a575669f4963a63889652ee338a5 2023-03-23 19:54:10,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9f2e68fcd6fa4d3a86a636d7c1a4d378 2023-03-23 19:54:10,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fcb62e5c2834404cbf6b5c1f29b14f47 2023-03-23 19:54:10,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec0b4d5b077e421483b539f10e4244b4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec0b4d5b077e421483b539f10e4244b4 2023-03-23 19:54:10,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ef1ddb04a2784942811437233761461f 2023-03-23 19:54:10,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/e59c11e8f8e942fd87394d4bb51f2139 2023-03-23 19:54:10,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b6fef5744d77448aa3684b2bcfcfb3dd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b6fef5744d77448aa3684b2bcfcfb3dd 2023-03-23 19:54:10,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/9d97c7868ced4edbb34639dd11f4d6b4 2023-03-23 19:54:10,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/f172aae99a4049e79e6cc231eff193d3 2023-03-23 19:54:10,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/1524b1a3697b4e0abe4aa76f14e160d0 2023-03-23 19:54:10,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/2191c61a06784573bf9e5eebaab59186 2023-03-23 19:54:10,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/648833f479ec446b9289a1cf1206b535 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/648833f479ec446b9289a1cf1206b535 2023-03-23 19:54:10,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b3a128e67cf94a61a2cb51e411cc7194 2023-03-23 19:54:10,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/eb4cc369556b4d2cb30f25b22af53aa6 2023-03-23 19:54:10,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/3f63204356d647d993f52d9134c9313a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/3f63204356d647d993f52d9134c9313a 2023-03-23 19:54:10,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ec6dc2f817c544aea4853f6caf8e28fc 2023-03-23 19:54:10,067 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/844e41085f234b7ebda1458d58f376bb 2023-03-23 19:54:10,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea44e72997754d1a82de479fd9d9344e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea44e72997754d1a82de479fd9d9344e 2023-03-23 19:54:10,069 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/ea57b74aecf443589d9d884c7f6b9e56 2023-03-23 19:54:10,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b1ac8e768fb946f4a7141fe9d7d58b16 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b1ac8e768fb946f4a7141fe9d7d58b16 2023-03-23 19:54:10,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/791d81a2209d4328afee2562542fe746 2023-03-23 19:54:10,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/fdd138091114430cbc067095ea628294 2023-03-23 19:54:10,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/b77371be78754e2fbc9113e7f28bafee 2023-03-23 19:54:10,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/862eb1075bf04356a08c523a09ce1bde 2023-03-23 19:54:10,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/75e942f12ecb4ce5afc0b4b9a263b549, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/dbf96f9bc1c949468bd53f9f48fdd8aa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f7ff8eb7c0324a1891c9ecc16f258d50, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/76fb3e0644574910b0f23c6435280ea1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6ae6be1d09204a919223b3c3a68f8e16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/bd8c5f9d2d1b49a6972587ec49938fea, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/fb0c8d0d942f41eaad906e0c74d0d41c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6a1dcb21e2d54add849d3544b40381d6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d527063a2ec0479cac26412c569fea84, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d961ca22898049b38f88f53053af1f2f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846] to archive 2023-03-23 19:54:10,101 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:10,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/295da66a285649a2b43e77ec87b3909f 2023-03-23 19:54:10,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d1f27a295fc4e98bb6c52365eda5635 2023-03-23 19:54:10,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/75e942f12ecb4ce5afc0b4b9a263b549 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/75e942f12ecb4ce5afc0b4b9a263b549 2023-03-23 19:54:10,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9717d1300b7546e7ad160d21e13ba11d 2023-03-23 19:54:10,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8d8013c83710439a99351bd6618813c7 2023-03-23 19:54:10,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ac0049dfc1c345a199ec49bdab6d17f5 2023-03-23 19:54:10,109 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/dbf96f9bc1c949468bd53f9f48fdd8aa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/dbf96f9bc1c949468bd53f9f48fdd8aa 2023-03-23 19:54:10,110 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9e3287b4bae7478eafd19857f74fb74a 2023-03-23 19:54:10,111 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/02960f24372047a796c77323b23c5c00 2023-03-23 19:54:10,112 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/72084c8ff7f54c568b7098e84dbbcf0b 2023-03-23 19:54:10,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f7ff8eb7c0324a1891c9ecc16f258d50 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f7ff8eb7c0324a1891c9ecc16f258d50 2023-03-23 19:54:10,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/12d1d957b28c4bada42ceca816e6abc0 2023-03-23 19:54:10,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/241ea27918bd408fb3f2953183d31f81 2023-03-23 19:54:10,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/76fb3e0644574910b0f23c6435280ea1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/76fb3e0644574910b0f23c6435280ea1 2023-03-23 19:54:10,117 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ab53435bfb8d4fc6940c24f9884170e2 2023-03-23 19:54:10,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/60962d12e20f4758aaaa6bcaa20bcbf1 2023-03-23 19:54:10,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/429d10784102417499adea11c3dabd2b 2023-03-23 19:54:10,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/ef53be9d53b347b08d2cad989ea5ae47 2023-03-23 19:54:10,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6ae6be1d09204a919223b3c3a68f8e16 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6ae6be1d09204a919223b3c3a68f8e16 2023-03-23 19:54:10,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/eaed619f79ef463d823e993f771a69ed 2023-03-23 19:54:10,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e6a8fdfb9a3d4a1e934e0485273bc8a4 2023-03-23 19:54:10,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/bd8c5f9d2d1b49a6972587ec49938fea to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/bd8c5f9d2d1b49a6972587ec49938fea 2023-03-23 19:54:10,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5dba76e04bb941a880c41a9bb10614a1 2023-03-23 19:54:10,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/8a9f05bda2ba414d8edab0edc3386f89 2023-03-23 19:54:10,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/668dd9f4b7c14612b6fc4ecfe02b1822 2023-03-23 19:54:10,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59e95f15cf1f4fa7bfa4256de194d3ea 2023-03-23 19:54:10,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/fb0c8d0d942f41eaad906e0c74d0d41c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/fb0c8d0d942f41eaad906e0c74d0d41c 2023-03-23 19:54:10,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/e1962871bc9347c48029b6b3fd253c5e 2023-03-23 19:54:10,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/81731046a1ba41e68cc872e373ee87b5 2023-03-23 19:54:10,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6a1dcb21e2d54add849d3544b40381d6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6a1dcb21e2d54add849d3544b40381d6 2023-03-23 19:54:10,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6f91d99700fa4d238d1c28bda1be85b6 2023-03-23 19:54:10,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/50cf8aecc92f45f5bb6e4a5dcbdb1abb 2023-03-23 19:54:10,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d527063a2ec0479cac26412c569fea84 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d527063a2ec0479cac26412c569fea84 2023-03-23 19:54:10,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/5c56dc2a7ac4486b9bb79e6cd8ef8e73 2023-03-23 19:54:10,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/7a0059dd78f24b91b6c7acffacaab943 2023-03-23 19:54:10,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d961ca22898049b38f88f53053af1f2f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/d961ca22898049b38f88f53053af1f2f 2023-03-23 19:54:10,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/59c4dc7c06d5488cb29c2068bf4e355b 2023-03-23 19:54:10,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/a1e21048ec74473a81f8fe654da33908 2023-03-23 19:54:10,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/6107c1d535c9439bb0f92a353e8973a8 2023-03-23 19:54:10,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/f4d4f40a04b844b7ae8691c27ba0f846 2023-03-23 19:54:10,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5ebc6c35531c4f8cba268e3d250a22e9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a22626c3c04b43368b0927f4a231ba55, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d724e8476b364a3abf1beea9539de2e6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/05d8c51ce420459c9fcc13d10d0fb6ae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/781c15e1e6934a8a913fd60251581960, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/4bcc2ee1346e48f29a6ef4b0c95562ec, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9c6a4cbeb58746d385a26a1100b72041, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7f15f4d75ffe48c9b2a495cc3af1a253, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/503375dec5be42df999dfb857cf5ee49, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/db98471ae6cc45dca83535d9dd605583, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef] to archive 2023-03-23 19:54:10,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:10,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/c977440f1b3d4b71ad527f49c207bdf4 2023-03-23 19:54:10,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/831df8b0df474ae9b45f97055d2c855c 2023-03-23 19:54:10,152 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5ebc6c35531c4f8cba268e3d250a22e9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5ebc6c35531c4f8cba268e3d250a22e9 2023-03-23 19:54:10,153 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/bc4729a9280643698c1b5950985ed62b 2023-03-23 19:54:10,154 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d501befee1f0413dab0ccdcfeee18fad 2023-03-23 19:54:10,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/e1edfeed451540e3b69e8212c9360b74 2023-03-23 19:54:10,156 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a22626c3c04b43368b0927f4a231ba55 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a22626c3c04b43368b0927f4a231ba55 2023-03-23 19:54:10,157 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/43d891909a4e40feb0c7b910a8e23831 2023-03-23 19:54:10,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7c67a6d6f0d142709d92c94ae0ccbc10 2023-03-23 19:54:10,159 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/ae7c30c37d4a451bb0cd85005403df93 2023-03-23 19:54:10,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d724e8476b364a3abf1beea9539de2e6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d724e8476b364a3abf1beea9539de2e6 2023-03-23 19:54:10,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a6b859e33f8d4dfcb3a8ae08dd551d19 2023-03-23 19:54:10,163 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a3fcec2a180441478f222c13d7da3c29 2023-03-23 19:54:10,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/05d8c51ce420459c9fcc13d10d0fb6ae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/05d8c51ce420459c9fcc13d10d0fb6ae 2023-03-23 19:54:10,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/1a711dfd66014405b5d46753a9296244 2023-03-23 19:54:10,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d185eed418e34696a9a8992cf1410389 2023-03-23 19:54:10,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/5313a75d9ae947c99ec1f7dbf1695f1f 2023-03-23 19:54:10,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/90840402fee8401583a299b515c5a543 2023-03-23 19:54:10,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/781c15e1e6934a8a913fd60251581960 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/781c15e1e6934a8a913fd60251581960 2023-03-23 19:54:10,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a67c66bce1f44bba8365dd1c1ca0dba8 2023-03-23 19:54:10,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/a142c37b18f44a2b83d6d42794ff9586 2023-03-23 19:54:10,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/4bcc2ee1346e48f29a6ef4b0c95562ec to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/4bcc2ee1346e48f29a6ef4b0c95562ec 2023-03-23 19:54:10,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/83632264ddfb40ba955c9f75f8dfe209 2023-03-23 19:54:10,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/32925b3056fd4ab1a651b84ba3a92d7a 2023-03-23 19:54:10,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b5b5638f88ee4244954ec7f5ca422dce 2023-03-23 19:54:10,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9273680ccb7445f2b0b7d73757725fc4 2023-03-23 19:54:10,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9c6a4cbeb58746d385a26a1100b72041 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9c6a4cbeb58746d385a26a1100b72041 2023-03-23 19:54:10,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9a4dfe2c68a14854af1a347e4bf9e12a 2023-03-23 19:54:10,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/93d4ee317a144a60912c3d0f56d56d11 2023-03-23 19:54:10,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7f15f4d75ffe48c9b2a495cc3af1a253 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7f15f4d75ffe48c9b2a495cc3af1a253 2023-03-23 19:54:10,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/8f7f2c0a08a1498c83807b6c1a30061c 2023-03-23 19:54:10,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/f9a3da868417499aa63a1906a0ff49ee 2023-03-23 19:54:10,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/503375dec5be42df999dfb857cf5ee49 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/503375dec5be42df999dfb857cf5ee49 2023-03-23 19:54:10,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/b8d66ab927af47fc8ae2a69ee10421ab 2023-03-23 19:54:10,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/d5f5f08c5e054dc8979806a5a1c24d70 2023-03-23 19:54:10,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/db98471ae6cc45dca83535d9dd605583 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/db98471ae6cc45dca83535d9dd605583 2023-03-23 19:54:10,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/77818b35274f4b83b7a98fc55679aa5e 2023-03-23 19:54:10,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/eb6af98b590d4ae0a95076f4a8f5ebff 2023-03-23 19:54:10,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/928b6e3dd40940f6a8b0dad48f96eb2c 2023-03-23 19:54:10,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/7811acd08072463b8695470c7129b6ef 2023-03-23 19:54:10,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/recovered.edits/830.seqid, newMaxSeqId=830, maxSeqId=1 2023-03-23 19:54:10,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e. 2023-03-23 19:54:10,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 01acdc2ff8c96b2160b74ed8a9a21f8e: 2023-03-23 19:54:10,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:10,210 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=01acdc2ff8c96b2160b74ed8a9a21f8e, regionState=CLOSED 2023-03-23 19:54:10,210 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601250210"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601250210"}]},"ts":"1679601250210"} 2023-03-23 19:54:10,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-23 19:54:10,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 01acdc2ff8c96b2160b74ed8a9a21f8e, server=jenkins-hbase4.apache.org,34277,1679601218015 in 1.6180 sec 2023-03-23 19:54:10,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-23 19:54:10,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=01acdc2ff8c96b2160b74ed8a9a21f8e, UNASSIGN in 1.6230 sec 2023-03-23 19:54:10,216 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601250216"}]},"ts":"1679601250216"} 2023-03-23 19:54:10,217 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:54:10,218 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:54:10,222 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6420 sec 2023-03-23 19:54:10,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-23 19:54:10,691 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-23 19:54:10,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:54:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,703 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,705 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-23 19:54:10,709 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:10,712 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/recovered.edits] 2023-03-23 19:54:10,715 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/31bdc8a85c584251ada3c429c2a130c0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/31bdc8a85c584251ada3c429c2a130c0 2023-03-23 19:54:10,717 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/d0ae5cf005e740a1b6d57a5c6dd9a735 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/A/d0ae5cf005e740a1b6d57a5c6dd9a735 2023-03-23 19:54:10,719 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9cb43f80269442dd9d1afa2e2489cbd6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/9cb43f80269442dd9d1afa2e2489cbd6 2023-03-23 19:54:10,720 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/c193cac4955545cdb9ac70c99ee13cf8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/B/c193cac4955545cdb9ac70c99ee13cf8 2023-03-23 19:54:10,722 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9472703abf3c4f04918ff2928ef72074 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9472703abf3c4f04918ff2928ef72074 2023-03-23 19:54:10,724 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9743c2e3b25a486eb71bc16f2bc8bc4e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/C/9743c2e3b25a486eb71bc16f2bc8bc4e 2023-03-23 19:54:10,727 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/recovered.edits/830.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e/recovered.edits/830.seqid 2023-03-23 19:54:10,727 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/01acdc2ff8c96b2160b74ed8a9a21f8e 2023-03-23 19:54:10,727 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:54:10,729 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,749 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:54:10,791 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:54:10,793 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,793 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:54:10,793 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601250793"}]},"ts":"9223372036854775807"} 2023-03-23 19:54:10,796 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:54:10,796 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 01acdc2ff8c96b2160b74ed8a9a21f8e, NAME => 'TestAcidGuarantees,,1679601222354.01acdc2ff8c96b2160b74ed8a9a21f8e.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:54:10,796 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:54:10,796 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601250796"}]},"ts":"9223372036854775807"} 2023-03-23 19:54:10,798 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:54:10,801 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 105 msec 2023-03-23 19:54:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-23 19:54:10,807 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-23 19:54:10,823 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=232 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6185653f-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1289081537_15 at /127.0.0.1:41514 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6185653f-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6185653f-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6185653f-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34277-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=578 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 327) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5630 (was 6277) 2023-03-23 19:54:10,834 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=578, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=170, AvailableMemoryMB=5629 2023-03-23 19:54:10,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:54:10,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:54:10,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:10,838 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:54:10,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-23 19:54:10,839 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:54:10,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:10,841 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:10,841 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 empty. 2023-03-23 19:54:10,842 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:10,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:54:10,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:11,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:11,249 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:54:11,265 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2301f05dcd8098430ba2e76ffa6a3bb0, NAME => 'TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:54:11,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2301f05dcd8098430ba2e76ffa6a3bb0, disabling compactions & flushes 2023-03-23 19:54:11,671 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. after waiting 0 ms 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:11,671 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:11,671 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:11,674 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:54:11,675 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601251674"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601251674"}]},"ts":"1679601251674"} 2023-03-23 19:54:11,676 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:54:11,676 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:54:11,677 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601251677"}]},"ts":"1679601251677"} 2023-03-23 19:54:11,678 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:54:11,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, ASSIGN}] 2023-03-23 19:54:11,683 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, ASSIGN 2023-03-23 19:54:11,684 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:54:11,835 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:11,835 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601251835"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601251835"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601251835"}]},"ts":"1679601251835"} 2023-03-23 19:54:11,837 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:11,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:11,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:11,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2301f05dcd8098430ba2e76ffa6a3bb0, NAME => 'TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:54:11,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:11,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:54:11,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:11,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:11,994 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:11,996 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A 2023-03-23 19:54:11,996 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A 2023-03-23 19:54:11,996 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:11,996 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName A 2023-03-23 19:54:11,997 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:11,997 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:11,998 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B 2023-03-23 19:54:11,998 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B 2023-03-23 19:54:11,999 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:11,999 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName B 2023-03-23 19:54:11,999 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:12,000 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:12,001 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C 2023-03-23 19:54:12,001 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C 2023-03-23 19:54:12,001 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:12,001 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName C 2023-03-23 19:54:12,002 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:12,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:12,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:12,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:12,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:54:12,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:12,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:54:12,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2301f05dcd8098430ba2e76ffa6a3bb0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72570810, jitterRate=0.08138933777809143}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:54:12,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:12,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., pid=18, masterSystemTime=1679601251988 2023-03-23 19:54:12,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:12,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:12,011 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:12,011 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601252011"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601252011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601252011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601252011"}]},"ts":"1679601252011"} 2023-03-23 19:54:12,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-23 19:54:12,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 in 175 msec 2023-03-23 19:54:12,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-23 19:54:12,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, ASSIGN in 333 msec 2023-03-23 19:54:12,016 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:54:12,016 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601252016"}]},"ts":"1679601252016"} 2023-03-23 19:54:12,017 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:54:12,019 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:54:12,020 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1830 sec 2023-03-23 19:54:12,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-23 19:54:12,944 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-23 19:54:12,945 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:12,952 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:12,953 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:12,957 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37370, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:12,958 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:54:12,962 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:54:12,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:54:12,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:13,389 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:54:13,390 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:54:13,390 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:54:13,394 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-23 19:54:13,406 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, REOPEN/MOVE}] 2023-03-23 19:54:13,406 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, REOPEN/MOVE 2023-03-23 19:54:13,407 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:13,407 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601253407"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601253407"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601253407"}]},"ts":"1679601253407"} 2023-03-23 19:54:13,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:13,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2301f05dcd8098430ba2e76ffa6a3bb0, disabling compactions & flushes 2023-03-23 19:54:13,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. after waiting 0 ms 2023-03-23 19:54:13,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-23 19:54:13,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:13,565 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2301f05dcd8098430ba2e76ffa6a3bb0 to self. 2023-03-23 19:54:13,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,567 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=CLOSED 2023-03-23 19:54:13,568 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601253567"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601253567"}]},"ts":"1679601253567"} 2023-03-23 19:54:13,570 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-23 19:54:13,571 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 in 161 msec 2023-03-23 19:54:13,571 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=true 2023-03-23 19:54:13,721 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:13,722 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601253721"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601253721"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601253721"}]},"ts":"1679601253721"} 2023-03-23 19:54:13,723 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:13,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2301f05dcd8098430ba2e76ffa6a3bb0, NAME => 'TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:54:13,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:54:13,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,881 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,882 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A 2023-03-23 19:54:13,882 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A 2023-03-23 19:54:13,883 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:13,885 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName A 2023-03-23 19:54:13,887 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:13,887 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,888 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B 2023-03-23 19:54:13,888 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B 2023-03-23 19:54:13,888 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:13,889 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName B 2023-03-23 19:54:13,889 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:13,889 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,890 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C 2023-03-23 19:54:13,890 DEBUG [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C 2023-03-23 19:54:13,890 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:13,890 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2301f05dcd8098430ba2e76ffa6a3bb0 columnFamilyName C 2023-03-23 19:54:13,891 INFO [StoreOpener-2301f05dcd8098430ba2e76ffa6a3bb0-1] regionserver.HStore(310): Store=2301f05dcd8098430ba2e76ffa6a3bb0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:13,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:54:13,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:13,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2301f05dcd8098430ba2e76ffa6a3bb0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65337294, jitterRate=-0.026398450136184692}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:54:13,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:13,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., pid=23, masterSystemTime=1679601253875 2023-03-23 19:54:13,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:13,900 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:13,900 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601253900"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601253900"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601253900"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601253900"}]},"ts":"1679601253900"} 2023-03-23 19:54:13,903 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-23 19:54:13,903 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 in 178 msec 2023-03-23 19:54:13,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-23 19:54:13,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, REOPEN/MOVE in 497 msec 2023-03-23 19:54:13,908 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-23 19:54:13,908 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-03-23 19:54:13,910 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 936 msec 2023-03-23 19:54:13,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-23 19:54:13,920 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,925 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,926 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,930 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,930 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,934 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,934 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,940 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,940 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,946 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,946 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,950 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,950 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,954 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,955 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,959 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,960 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:13,965 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:13,975 DEBUG [hconnection-0x3254ccd3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,975 DEBUG [hconnection-0x21cd4522-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,976 DEBUG [hconnection-0x2294e7ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,976 DEBUG [hconnection-0xc480d09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,978 DEBUG [hconnection-0x25adaa4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,978 DEBUG [hconnection-0x37777c34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,979 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,979 DEBUG [hconnection-0x188d4d0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,979 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48532, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,980 DEBUG [hconnection-0x151c3500-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,982 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,984 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48514, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,984 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,984 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48544, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,984 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48546, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,985 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48558, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:13,986 DEBUG [hconnection-0x3631d1a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:13,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:13,996 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48574, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:14,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,008 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:14,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:54:14,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:14,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:14,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:14,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:14,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:14,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:14,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601314036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601314037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601314038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601314038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25dbab63[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,046 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,046 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601314043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,049 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,049 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,054 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,054 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,054 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,054 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,054 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,059 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,062 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,071 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,071 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,071 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601314140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601314142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601314143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601314143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601314149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,172 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,188 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,200 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,208 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,208 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,208 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,208 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7413fa22[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,220 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,220 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,220 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,220 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,220 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,220 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,227 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,227 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,227 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,233 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,243 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,243 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,343 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601314344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601314346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601314347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601314347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,352 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601314350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,356 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,379 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,379 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,380 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,380 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,380 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,380 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,382 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513a1312[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,387 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,393 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,393 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,398 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,405 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,405 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,505 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,505 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303231ab22e54879d443598a3a571813d0ad4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231ab22e54879d443598a3a571813d0ad4 2023-03-23 19:54:14,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,520 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,522 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,523 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,523 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,543 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,543 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,543 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,543 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,545 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7def63[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,553 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,556 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,556 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,564 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,564 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601314651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601314651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601314652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601314652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601314656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,664 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,672 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,694 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2705d75c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,707 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,707 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,717 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,717 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,718 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,718 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,819 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,823 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,824 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,826 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,826 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,826 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,826 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4caceb74[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,839 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,839 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,842 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,844 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,852 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,852 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,931 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/eae3ff906f764758bcb0e07d9b55bbeb 2023-03-23 19:54:14,952 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:14,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:14,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:14,964 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:14,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:14,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:14,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/49658aed4ba247f6ac2f8139fb70b0bd 2023-03-23 19:54:14,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,966 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:14,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:14,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:14,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:14,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:14,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:14,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,977 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d8fd370[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:14,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:14,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:14,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:14,982 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,982 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:14,983 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:14,987 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:14,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:14,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:14,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:14,987 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:14,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:14,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:14,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:14,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:14,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:14,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:14,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:14,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:14,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:14,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:14,998 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:14,998 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:14,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:15,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/2c9b43cf284e43b89485f0ba629b880a 2023-03-23 19:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/eae3ff906f764758bcb0e07d9b55bbeb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb 2023-03-23 19:54:15,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb, entries=250, sequenceid=18, filesize=38.7 K 2023-03-23 19:54:15,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/49658aed4ba247f6ac2f8139fb70b0bd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd 2023-03-23 19:54:15,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd, entries=150, sequenceid=18, filesize=11.6 K 2023-03-23 19:54:15,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/2c9b43cf284e43b89485f0ba629b880a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a 2023-03-23 19:54:15,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a, entries=150, sequenceid=18, filesize=11.6 K 2023-03-23 19:54:15,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1025ms, sequenceid=18, compaction requested=false 2023-03-23 19:54:15,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:15,098 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:15,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:15,102 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:15,102 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:15,102 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:15,102 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:15,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:15,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:15,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,104 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:15,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:15,105 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,105 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:15,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:15,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:15,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:15,124 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:15,124 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:15,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:15,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:15,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:15,167 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032371cbc30b1f044a4d8afbdaffb8251dc5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032371cbc30b1f044a4d8afbdaffb8251dc5 2023-03-23 19:54:15,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601315166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601315167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601315169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601315169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601315173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,186 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/78c4c04d899f44939d98913af7e5431f 2023-03-23 19:54:15,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601315276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601315277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601315276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601315277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601315277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601315481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601315481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601315481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601315481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601315481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,603 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/b1e2832d566c40339b57f93db29c7da3 2023-03-23 19:54:15,753 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:54:15,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601315784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601315785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601315785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601315786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:15,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:15,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601315786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,015 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ddb7e6edb98348c286e00a1178d8cdae 2023-03-23 19:54:16,021 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/78c4c04d899f44939d98913af7e5431f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f 2023-03-23 19:54:16,029 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f, entries=150, sequenceid=37, filesize=25.1 K 2023-03-23 19:54:16,030 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/b1e2832d566c40339b57f93db29c7da3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3 2023-03-23 19:54:16,037 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3, entries=150, sequenceid=37, filesize=11.6 K 2023-03-23 19:54:16,038 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ddb7e6edb98348c286e00a1178d8cdae as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae 2023-03-23 19:54:16,042 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae, entries=150, sequenceid=37, filesize=11.6 K 2023-03-23 19:54:16,043 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 919ms, sequenceid=37, compaction requested=false 2023-03-23 19:54:16,043 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:16,044 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:16,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:16,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d392b55[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:16,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:16,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:16,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:16,049 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:16,052 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,052 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:16,054 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:16,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:16,056 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:16,056 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:16,056 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:16,059 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:16,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:16,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:16,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:16,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:16,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:16,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,068 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:16,068 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:16,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:16,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:16,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:16,168 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:16,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:16,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:16,175 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:16,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:16,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:16,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:16,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:16,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:16,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:16,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:16,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:16,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:16,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:16,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:16,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:16,181 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:16,181 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:16,181 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:54:16,181 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:16,181 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:16,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:16,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:16,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601316307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601316308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601316309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601316311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601316313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601316414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601316416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601316417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601316417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601316418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,617 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323947d776ec6b84f0bb83b53850e71df5f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323947d776ec6b84f0bb83b53850e71df5f 2023-03-23 19:54:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601316618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601316618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601316622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601316623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601316623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601316919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601316921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601316923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601316926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:16,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601316926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,023 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/991351fd214b40948323f8d72e0a851d 2023-03-23 19:54:17,038 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/3ae2326465de4cc8b6c06ba72227e653 2023-03-23 19:54:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:17,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601317425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:17,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601317425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:17,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601317430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:17,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601317433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:17,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601317440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,454 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/5c095fe6ecec412aa824d9f7cc2c06e9 2023-03-23 19:54:17,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/991351fd214b40948323f8d72e0a851d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d 2023-03-23 19:54:17,477 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d, entries=150, sequenceid=50, filesize=25.1 K 2023-03-23 19:54:17,478 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/3ae2326465de4cc8b6c06ba72227e653 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653 2023-03-23 19:54:17,485 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653, entries=150, sequenceid=50, filesize=11.6 K 2023-03-23 19:54:17,486 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/5c095fe6ecec412aa824d9f7cc2c06e9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9 2023-03-23 19:54:17,504 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9, entries=150, sequenceid=50, filesize=11.6 K 2023-03-23 19:54:17,505 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1324ms, sequenceid=50, compaction requested=true 2023-03-23 19:54:17,505 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:17,505 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:17,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,507 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740fbc99[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:17,509 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:17,509 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,511 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:17,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:17,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:17,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:17,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:17,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:17,514 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:17,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:17,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:17,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:17,514 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:17,515 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,515 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,516 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:17,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:17,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:17,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:17,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:17,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:17,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:17,521 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:17,521 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:17,621 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:17,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:17,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:17,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:17,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:17,625 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:17,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:17,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,626 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:17,628 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:17,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:17,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:17,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:17,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:17,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:17,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:17,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:17,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:17,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:17,631 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:54:17,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:17,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:17,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:17,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:17,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:17,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:18,047 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323eb31651ca4ea48f2b1a6a197c66f420a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb31651ca4ea48f2b1a6a197c66f420a 2023-03-23 19:54:18,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:18,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:18,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601318447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601318447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601318448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601318448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601318449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,454 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/b01e381e55bd4fd0914af0c94c2c4298 2023-03-23 19:54:18,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601318550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601318551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601318551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601318754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601318755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601318755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:18,866 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/5898cbf146dd4a2e8fa8d0218ebdc1a3 2023-03-23 19:54:19,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601319061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601319062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601319062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,277 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/f2ce91f4b8124e79b148dc998dbb5e48 2023-03-23 19:54:19,283 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/b01e381e55bd4fd0914af0c94c2c4298 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298 2023-03-23 19:54:19,288 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298, entries=150, sequenceid=69, filesize=25.1 K 2023-03-23 19:54:19,289 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/5898cbf146dd4a2e8fa8d0218ebdc1a3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3 2023-03-23 19:54:19,294 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3, entries=150, sequenceid=69, filesize=11.6 K 2023-03-23 19:54:19,295 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/f2ce91f4b8124e79b148dc998dbb5e48 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48 2023-03-23 19:54:19,300 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48, entries=150, sequenceid=69, filesize=11.6 K 2023-03-23 19:54:19,301 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1670ms, sequenceid=69, compaction requested=true 2023-03-23 19:54:19,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:19,301 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:19,305 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:19,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:19,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f530e27[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:19,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:19,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:19,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:19,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:19,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:19,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:19,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:19,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:19,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:19,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:19,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:19,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:19,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,322 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:19,322 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:19,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:19,423 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:19,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:19,428 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:19,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:19,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:19,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:19,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:19,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:19,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:19,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:19,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:19,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:19,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:19,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:19,437 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:19,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:19,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:19,438 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:19,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:19,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:19,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601319596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601319596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601319599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601319700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601319700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601319705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,870 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303239f46ebd07f22489fa3d8914579158266 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f46ebd07f22489fa3d8914579158266 2023-03-23 19:54:19,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601319905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601319905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601319910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601320208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601320209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601320213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,276 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/1f13bcec85114cb4866b726fd7793dd5 2023-03-23 19:54:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601320464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,466 DEBUG [Thread-958] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601320466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,470 DEBUG [Thread-956] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:20,692 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/3eaa188d9e3b4832a7d42cb3afd1d78b 2023-03-23 19:54:20,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601320712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601320716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:20,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601320717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,106 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/e9702b9df4584e4681f57fc85f78a3f7 2023-03-23 19:54:21,111 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/1f13bcec85114cb4866b726fd7793dd5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5 2023-03-23 19:54:21,115 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5, entries=150, sequenceid=82, filesize=25.1 K 2023-03-23 19:54:21,116 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/3eaa188d9e3b4832a7d42cb3afd1d78b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b 2023-03-23 19:54:21,122 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b, entries=150, sequenceid=82, filesize=11.6 K 2023-03-23 19:54:21,123 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/e9702b9df4584e4681f57fc85f78a3f7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7 2023-03-23 19:54:21,128 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7, entries=150, sequenceid=82, filesize=11.6 K 2023-03-23 19:54:21,129 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1692ms, sequenceid=82, compaction requested=true 2023-03-23 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:21,129 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:21,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,134 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:21,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:21,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c7452ee[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:21,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:21,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,138 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:21,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:21,140 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:21,141 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:21,145 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:21,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:21,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:21,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:21,145 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:21,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:21,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:21,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,155 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:21,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:21,156 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:21,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:21,156 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:21,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:21,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:21,256 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:21,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:21,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:21,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:21,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:21,260 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:21,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:21,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:21,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:21,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:21,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:21,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:21,263 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,263 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:21,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:21,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:21,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:21,264 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:21,264 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:21,264 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:21,683 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323d165db2efba24b9e86bfdbd52db3edb6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d165db2efba24b9e86bfdbd52db3edb6 2023-03-23 19:54:21,688 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/195da42c3e5d4227877503ec27a4f463 2023-03-23 19:54:21,703 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e1af1babfdb54df4a291deadac33ed82 2023-03-23 19:54:21,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:21,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:21,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601321730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601321731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601321734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601321835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601321837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:21,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601321838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601322038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601322039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601322042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/53dab775e2ab4f4ea8da3ddc8d893f9a 2023-03-23 19:54:22,124 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/195da42c3e5d4227877503ec27a4f463 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463 2023-03-23 19:54:22,130 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463, entries=150, sequenceid=101, filesize=25.1 K 2023-03-23 19:54:22,131 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e1af1babfdb54df4a291deadac33ed82 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82 2023-03-23 19:54:22,136 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82, entries=150, sequenceid=101, filesize=11.6 K 2023-03-23 19:54:22,137 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/53dab775e2ab4f4ea8da3ddc8d893f9a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a 2023-03-23 19:54:22,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a, entries=150, sequenceid=101, filesize=11.6 K 2023-03-23 19:54:22,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 881ms, sequenceid=101, compaction requested=true 2023-03-23 19:54:22,146 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:22,146 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:22,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5823afb6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:22,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:22,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:22,154 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:22,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:22,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:22,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:22,159 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:22,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:22,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:22,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:22,159 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:22,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:22,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:22,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:22,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,169 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:22,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:22,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:22,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:22,172 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:22,172 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:22,172 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:22,272 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:22,276 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:22,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:22,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:22,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:22,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:22,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,279 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:22,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:22,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:22,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:22,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:22,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:22,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:22,280 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:22,280 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:22,281 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:22,281 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:22,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:22,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:22,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601322361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601322362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601322362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601322465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601322466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601322466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601322669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601322669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601322670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,695 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303238a601cd4b8d74dc7bd3927c8699d890a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238a601cd4b8d74dc7bd3927c8699d890a 2023-03-23 19:54:22,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601322972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601322972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:22,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601322973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,105 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/0e6843130466449ba0a3cc0e60103dc9 2023-03-23 19:54:23,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e5c5016d51df44ba9019d8cdfc7b611b 2023-03-23 19:54:23,129 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/7941e0d32f1040dca7f0dede170c7f21 2023-03-23 19:54:23,133 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/0e6843130466449ba0a3cc0e60103dc9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9 2023-03-23 19:54:23,137 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9, entries=150, sequenceid=114, filesize=25.1 K 2023-03-23 19:54:23,138 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e5c5016d51df44ba9019d8cdfc7b611b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b 2023-03-23 19:54:23,143 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b, entries=150, sequenceid=114, filesize=11.6 K 2023-03-23 19:54:23,143 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/7941e0d32f1040dca7f0dede170c7f21 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21 2023-03-23 19:54:23,147 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21, entries=150, sequenceid=114, filesize=11.6 K 2023-03-23 19:54:23,148 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 867ms, sequenceid=114, compaction requested=true 2023-03-23 19:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:23,148 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:23,149 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:23,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51c12b5f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:23,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:23,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,154 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,154 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,154 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:23,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:23,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:23,154 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:23,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:23,159 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:23,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:23,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:23,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:23,159 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:23,163 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,163 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,163 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:23,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:23,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,165 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:23,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:23,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:23,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:23,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:23,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:23,168 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:23,168 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:23,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:23,269 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:23,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:23,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:23,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:23,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:23,272 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:23,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:23,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:23,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:23,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:23,274 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:23,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:23,274 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,274 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:23,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:23,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:23,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:23,275 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:23,276 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:23,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:23,289 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303234e9428196fd54bd8abec99726022422c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234e9428196fd54bd8abec99726022422c 2023-03-23 19:54:23,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:23,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:23,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601323485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601323488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601323488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601323589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601323593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601323593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,698 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/0be0f190f1d84325bc6a29fb72dcacbd 2023-03-23 19:54:23,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601323793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601323797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:23,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:23,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601323798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601324095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601324103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601324104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,110 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/90fd6eaa6ef3476aa074c47d8595bf1b 2023-03-23 19:54:24,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48574 deadline: 1679601324479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,483 DEBUG [Thread-958] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:24,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48558 deadline: 1679601324502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,505 DEBUG [Thread-956] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:24,521 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/86a2ca598ebf4932b8c233733c4b7e0f 2023-03-23 19:54:24,526 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/0be0f190f1d84325bc6a29fb72dcacbd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd 2023-03-23 19:54:24,530 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd, entries=150, sequenceid=134, filesize=25.3 K 2023-03-23 19:54:24,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/90fd6eaa6ef3476aa074c47d8595bf1b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b 2023-03-23 19:54:24,536 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b, entries=150, sequenceid=134, filesize=11.8 K 2023-03-23 19:54:24,537 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/86a2ca598ebf4932b8c233733c4b7e0f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f 2023-03-23 19:54:24,552 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f, entries=150, sequenceid=134, filesize=11.8 K 2023-03-23 19:54:24,553 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1277ms, sequenceid=134, compaction requested=true 2023-03-23 19:54:24,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:24,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:24,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:24,556 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:24,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a42576[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:24,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:24,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,560 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:24,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:24,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:24,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:24,567 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:24,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:24,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:24,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:24,567 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:24,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:24,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:24,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,576 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:24,576 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:24,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:24,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:54:24,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:24,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:24,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:24,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:24,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:24,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:24,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601324639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601324640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601324640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,676 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:24,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:24,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:24,680 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:24,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:24,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:24,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,682 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:24,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:24,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:24,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:24,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:24,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:24,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,685 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5809f66e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:24,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:24,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:24,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:24,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:24,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:24,694 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:24,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:24,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:24,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:24,694 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:24,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,696 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,698 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:24,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:24,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,701 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:24,701 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601324748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601324753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601324753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,801 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:24,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:24,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:24,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:24,805 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:24,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:24,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:24,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,807 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,808 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,808 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:24,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:24,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:24,816 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,817 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:24,817 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:24,817 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:24,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d92acc6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:24,821 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:24,821 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,823 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:24,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:24,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:24,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:24,824 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:24,827 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:24,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:24,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:24,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:24,827 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:24,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,829 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,840 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:24,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,841 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:24,841 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:24,941 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:24,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:24,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:24,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:24,945 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:24,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:24,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:24,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:24,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:24,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:24,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:24,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,949 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:24,949 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:24,949 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:24,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:24,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601324949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,952 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d1e7f8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:24,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:24,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:24,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:24,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:24,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:24,961 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:24,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:24,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:24,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:24,961 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:24,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601324959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:24,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:24,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:24,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:24,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601324960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:24,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:24,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,972 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:24,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:24,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:24,972 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:24,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:24,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:24,973 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:24,973 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:24,973 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:24,973 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:24,973 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:24,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,028 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323efbcaad7caa24852ab7a9ee31986bd7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323efbcaad7caa24852ab7a9ee31986bd7b 2023-03-23 19:54:25,035 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/adcf21b1d915459a838351eb21860d77 2023-03-23 19:54:25,073 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:25,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:25,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:25,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:25,076 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:25,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:25,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:25,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:25,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:25,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:25,082 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:25,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e9e5a1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:25,083 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:25,083 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:25,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:25,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:25,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:25,090 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:25,090 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:25,090 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:25,090 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:25,090 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:25,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,093 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:25,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,096 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,096 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:25,097 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:25,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,197 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:25,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:25,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:25,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:25,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:25,200 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:25,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:25,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:25,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,202 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,202 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:25,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,204 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:25,205 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:25,205 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:25,206 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c4d714[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:25,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:25,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:25,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:25,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:25,211 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:25,213 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:25,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:25,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:25,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:25,214 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:25,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,216 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,218 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,222 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:25,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:25,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,223 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:25,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,223 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:25,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601325254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601325263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601325269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,323 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:25,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:25,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:25,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:25,326 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:25,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:25,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:25,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,329 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:25,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:25,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,331 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:25,331 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:25,331 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:25,334 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:25,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db8a3f6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:25,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:25,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:25,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:25,338 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:25,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:25,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:25,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:25,342 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:25,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:25,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:25,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:25,342 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:25,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,346 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,349 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:25,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,349 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:25,349 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,449 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:25,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:25,452 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,454 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:25,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:25,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:25,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:25,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:25,458 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@271c679e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:25,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:25,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:25,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/0d3b9aebbda4479292566926dd86ece0 2023-03-23 19:54:25,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:25,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:25,472 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:25,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:25,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:25,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:25,472 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:25,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:25,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ff7302baeea44ab1b37ce3c99ce7d89b 2023-03-23 19:54:25,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,479 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/adcf21b1d915459a838351eb21860d77 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77 2023-03-23 19:54:25,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77, entries=200, sequenceid=148, filesize=32.1 K 2023-03-23 19:54:25,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/0d3b9aebbda4479292566926dd86ece0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0 2023-03-23 19:54:25,492 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,494 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:25,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:25,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:25,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:25,495 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:25,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:25,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,496 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:25,496 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:25,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0, entries=150, sequenceid=148, filesize=11.8 K 2023-03-23 19:54:25,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ff7302baeea44ab1b37ce3c99ce7d89b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b 2023-03-23 19:54:25,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b, entries=150, sequenceid=148, filesize=11.8 K 2023-03-23 19:54:25,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 907ms, sequenceid=148, compaction requested=true 2023-03-23 19:54:25,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:25,510 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:25,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:25,510 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:25,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:25,511 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:25,516 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 252888 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:25,516 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/A is initiating minor compaction (all files) 2023-03-23 19:54:25,516 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/A in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,517 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=247.0 K 2023-03-23 19:54:25,517 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting eae3ff906f764758bcb0e07d9b55bbeb, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679601254001 2023-03-23 19:54:25,518 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107274 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:25,518 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/B is initiating minor compaction (all files) 2023-03-23 19:54:25,518 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 78c4c04d899f44939d98913af7e5431f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601254032 2023-03-23 19:54:25,518 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/B in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,518 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=104.8 K 2023-03-23 19:54:25,518 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 991351fd214b40948323f8d72e0a851d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601255158 2023-03-23 19:54:25,519 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 49658aed4ba247f6ac2f8139fb70b0bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679601254001 2023-03-23 19:54:25,519 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b01e381e55bd4fd0914af0c94c2c4298, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679601256304 2023-03-23 19:54:25,519 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 1f13bcec85114cb4866b726fd7793dd5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679601258446 2023-03-23 19:54:25,519 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b1e2832d566c40339b57f93db29c7da3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601254032 2023-03-23 19:54:25,520 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 195da42c3e5d4227877503ec27a4f463, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679601259586 2023-03-23 19:54:25,520 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3ae2326465de4cc8b6c06ba72227e653, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601255158 2023-03-23 19:54:25,520 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 0e6843130466449ba0a3cc0e60103dc9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679601261728 2023-03-23 19:54:25,520 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5898cbf146dd4a2e8fa8d0218ebdc1a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679601256304 2023-03-23 19:54:25,520 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 0be0f190f1d84325bc6a29fb72dcacbd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679601262357 2023-03-23 19:54:25,521 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3eaa188d9e3b4832a7d42cb3afd1d78b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679601258446 2023-03-23 19:54:25,521 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting adcf21b1d915459a838351eb21860d77, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:25,521 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e1af1babfdb54df4a291deadac33ed82, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679601259586 2023-03-23 19:54:25,522 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e5c5016d51df44ba9019d8cdfc7b611b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679601261728 2023-03-23 19:54:25,522 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 90fd6eaa6ef3476aa074c47d8595bf1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679601262357 2023-03-23 19:54:25,523 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 0d3b9aebbda4479292566926dd86ece0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:25,546 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#B#compaction#154 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:25,557 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#A#compaction#153 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:25,569 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/44afc49dc1cf410781945473f6fb79a8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/44afc49dc1cf410781945473f6fb79a8 2023-03-23 19:54:25,575 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/B of 2301f05dcd8098430ba2e76ffa6a3bb0 into 44afc49dc1cf410781945473f6fb79a8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:25,575 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:25,575 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/B, priority=7, startTime=1679601265510; duration=0sec 2023-03-23 19:54:25,576 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:25,576 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:25,578 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107274 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:25,578 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/C is initiating minor compaction (all files) 2023-03-23 19:54:25,578 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/C in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,578 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=104.8 K 2023-03-23 19:54:25,579 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2c9b43cf284e43b89485f0ba629b880a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679601254001 2023-03-23 19:54:25,579 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ddb7e6edb98348c286e00a1178d8cdae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601254032 2023-03-23 19:54:25,580 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5c095fe6ecec412aa824d9f7cc2c06e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679601255158 2023-03-23 19:54:25,580 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f2ce91f4b8124e79b148dc998dbb5e48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679601256304 2023-03-23 19:54:25,580 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e9702b9df4584e4681f57fc85f78a3f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679601258446 2023-03-23 19:54:25,581 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 53dab775e2ab4f4ea8da3ddc8d893f9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679601259586 2023-03-23 19:54:25,581 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7941e0d32f1040dca7f0dede170c7f21, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679601261728 2023-03-23 19:54:25,582 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 86a2ca598ebf4932b8c233733c4b7e0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679601262357 2023-03-23 19:54:25,582 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ff7302baeea44ab1b37ce3c99ce7d89b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:25,596 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#C#compaction#155 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:25,596 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:25,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:25,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:25,603 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/b13a2b270ea04366bafaea2b0a256417 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b13a2b270ea04366bafaea2b0a256417 2023-03-23 19:54:25,608 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/A of 2301f05dcd8098430ba2e76ffa6a3bb0 into b13a2b270ea04366bafaea2b0a256417(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:25,608 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:25,608 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/A, priority=7, startTime=1679601265510; duration=0sec 2023-03-23 19:54:25,608 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:25,619 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:25,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:25,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:25,620 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,620 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,621 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:25,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:25,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:25,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:25,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:25,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:25,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:25,629 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/6b4ec2e5d61e46548538dbee5da14b92 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6b4ec2e5d61e46548538dbee5da14b92 2023-03-23 19:54:25,633 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/C of 2301f05dcd8098430ba2e76ffa6a3bb0 into 6b4ec2e5d61e46548538dbee5da14b92(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:25,633 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:25,633 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/C, priority=7, startTime=1679601265510; duration=0sec 2023-03-23 19:54:25,633 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:25,636 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:25,636 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:25,636 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:25,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:25,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:25,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601325777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601325777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601325777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601325881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601325881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601325881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,051 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323034d977cebd44f27bcf32a17759cbaca to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323034d977cebd44f27bcf32a17759cbaca 2023-03-23 19:54:26,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601326085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601326085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601326085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601326387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48516 deadline: 1679601326387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601326389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,465 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/4cecbe1d78584185a2c70a0b068bd329 2023-03-23 19:54:26,498 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/9a190140993e4b4aaa3fc3da5a958bd0 2023-03-23 19:54:26,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601326893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48552 deadline: 1679601326893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:26,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48532 deadline: 1679601326894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,917 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/4da1bf9d055148759ceae427e3f523b1 2023-03-23 19:54:26,922 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/4cecbe1d78584185a2c70a0b068bd329 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329 2023-03-23 19:54:26,926 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329, entries=150, sequenceid=169, filesize=25.3 K 2023-03-23 19:54:26,927 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/9a190140993e4b4aaa3fc3da5a958bd0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0 2023-03-23 19:54:26,931 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0, entries=150, sequenceid=169, filesize=11.8 K 2023-03-23 19:54:26,932 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/4da1bf9d055148759ceae427e3f523b1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1 2023-03-23 19:54:26,936 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1, entries=150, sequenceid=169, filesize=11.8 K 2023-03-23 19:54:26,937 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1301ms, sequenceid=169, compaction requested=false 2023-03-23 19:54:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:26,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7155edc0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:26,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:26,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:26,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:26,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:26,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:26,949 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:26,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:26,949 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:26,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:26,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:26,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:26,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:26,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:26,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:26,956 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:26,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,956 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:26,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:26,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:26,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:26,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,056 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:27,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:27,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:27,059 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:27,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:27,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:27,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,062 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:27,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:27,063 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,063 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:27,063 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:27,063 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:27,063 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:27,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:27,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,078 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303239e1cf59a819a4b378463ac4f752df307 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239e1cf59a819a4b378463ac4f752df307 2023-03-23 19:54:27,088 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/f6c1ad3242f04d72825d55a9837c5d2d 2023-03-23 19:54:27,499 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/238de375080a4d60a7ede8e27fcca011 2023-03-23 19:54:27,509 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/a92400eb569e423ba60a8bd162a04f0c 2023-03-23 19:54:27,514 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/f6c1ad3242f04d72825d55a9837c5d2d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d 2023-03-23 19:54:27,519 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d, entries=150, sequenceid=185, filesize=25.3 K 2023-03-23 19:54:27,520 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/238de375080a4d60a7ede8e27fcca011 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011 2023-03-23 19:54:27,524 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011, entries=150, sequenceid=185, filesize=11.8 K 2023-03-23 19:54:27,525 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/a92400eb569e423ba60a8bd162a04f0c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c 2023-03-23 19:54:27,533 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c, entries=150, sequenceid=185, filesize=11.8 K 2023-03-23 19:54:27,534 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 471ms, sequenceid=185, compaction requested=true 2023-03-23 19:54:27,534 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:27,534 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:27,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:27,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b154e3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:27,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:27,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:27,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:27,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:27,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:27,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:27,555 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:27,555 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:27,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,563 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:27,563 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:27,663 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:27,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:27,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:27,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:27,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:27,667 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:27,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:27,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:27,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,669 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:27,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:27,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:27,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:27,671 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,671 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:27,671 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:27,671 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,673 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:27,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53534a6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:27,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:27,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,677 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:27,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:27,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:27,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:27,682 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:27,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:27,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:27,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:27,682 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:27,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,686 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,688 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:27,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:27,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,689 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:27,689 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:27,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,789 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:27,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:27,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:27,794 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:27,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:27,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:27,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,796 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:27,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:27,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:27,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:27,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:27,801 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:27,801 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:27,803 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e745c3c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:27,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:27,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,807 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,807 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:27,809 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:27,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:27,812 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:27,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:27,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:27,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:27,812 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:27,814 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,814 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:27,814 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:27,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,819 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:27,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,819 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:27,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:27,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:27,920 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:27,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:27,923 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:27,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:27,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:27,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,925 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:27,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:27,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:27,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:27,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:27,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:27,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:27,928 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-23 19:54:27,929 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:27,929 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,929 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:27,930 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,930 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:27,930 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:27,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:27,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679601267940, max timestamp=1679601267983Num uniques 859; 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679601267940, max timestamp=1679601267983Num uniques 859; 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679601267940, max timestamp=1679601267983Num uniques 859; 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-23 19:54:27,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-23 19:54:27,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601327993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:27,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601327993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601327998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601328099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601328099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601328101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601328323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601328323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601328323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,358 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323a241a94b17a84c268e1ea34a06920e00 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a241a94b17a84c268e1ea34a06920e00 2023-03-23 19:54:28,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601328625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601328626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:28,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601328626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:28,767 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/5ca30040795345ac88b9bd159dbcce44 2023-03-23 19:54:28,794 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/559bfc1cc6f6489e9fc8d829350f0b57 2023-03-23 19:54:29,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601329129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601329129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601329129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,205 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ab543750546d4f5b8a5444fb20c0b805 2023-03-23 19:54:29,211 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/5ca30040795345ac88b9bd159dbcce44 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44 2023-03-23 19:54:29,215 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44, entries=150, sequenceid=193, filesize=25.3 K 2023-03-23 19:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/559bfc1cc6f6489e9fc8d829350f0b57 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57 2023-03-23 19:54:29,221 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57, entries=100, sequenceid=193, filesize=9.4 K 2023-03-23 19:54:29,222 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/ab543750546d4f5b8a5444fb20c0b805 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805 2023-03-23 19:54:29,226 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805, entries=100, sequenceid=193, filesize=9.4 K 2023-03-23 19:54:29,227 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=181.14 KB/185490 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1299ms, sequenceid=193, compaction requested=true 2023-03-23 19:54:29,227 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:29,227 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:29,227 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:29,227 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:29,227 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,227 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:29,227 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:29,230 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:29,230 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50919bdf[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:29,239 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:29,239 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:29,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:29,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:29,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:29,248 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:29,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:29,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:29,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:29,248 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:29,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:29,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:29,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:29,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:29,255 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:29,255 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,255 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:29,255 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,255 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:29,255 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:29,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:29,355 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:29,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:29,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:29,359 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:29,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:29,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:29,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:29,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:29,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:29,360 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:29,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:29,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:29,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:29,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:29,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:29,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:29,362 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:29,362 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:29,362 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=181.14 KB heapSize=458.81 KB 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:29,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:29,404 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032385581c3c691a49cb9137e397cb160f47 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032385581c3c691a49cb9137e397cb160f47 2023-03-23 19:54:29,414 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/d60871d5a0be4239baca70ff638158da 2023-03-23 19:54:29,848 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/ae6961380de642599d1a6e20ea3dd95c 2023-03-23 19:54:30,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:30,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:30,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601330139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601330139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601330140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601330241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601330241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601330242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,340 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/899ac687e68a4f0ebe5ca14748c70c7b 2023-03-23 19:54:30,345 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/d60871d5a0be4239baca70ff638158da as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da 2023-03-23 19:54:30,349 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da, entries=150, sequenceid=223, filesize=25.3 K 2023-03-23 19:54:30,350 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/ae6961380de642599d1a6e20ea3dd95c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c 2023-03-23 19:54:30,354 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c, entries=150, sequenceid=223, filesize=11.8 K 2023-03-23 19:54:30,355 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/899ac687e68a4f0ebe5ca14748c70c7b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b 2023-03-23 19:54:30,365 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b, entries=150, sequenceid=223, filesize=11.8 K 2023-03-23 19:54:30,366 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~458.77 KB/469776, currentSize=20.13 KB/20610 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1004ms, sequenceid=223, compaction requested=true 2023-03-23 19:54:30,366 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:30,366 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:30,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:30,368 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:30,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb8cb62[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:30,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:30,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:30,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:30,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:30,378 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:30,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:30,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:30,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:30,378 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:30,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:30,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,385 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:30,385 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:30,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:30,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:54:30,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:30,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:30,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:30,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:30,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:30,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:30,472 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323acff883c0f39404fa2ec2d1bfacc315f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323acff883c0f39404fa2ec2d1bfacc315f 2023-03-23 19:54:30,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601330480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601330481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601330482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,485 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:30,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:30,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:30,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:30,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:30,489 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:30,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:30,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:30,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,491 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,491 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,491 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,491 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:30,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:30,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:30,492 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:30,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,496 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,496 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43045237[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:30,497 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:30,497 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,499 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:30,499 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,499 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:30,503 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:30,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:30,503 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:30,504 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,504 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,504 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:30,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:30,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,509 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:30,509 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,509 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601330585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601330585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601330586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,609 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:30,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:30,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:30,615 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:30,615 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,615 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:30,615 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:30,616 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,616 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,617 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,618 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,618 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:30,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:30,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:30,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:30,619 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:30,619 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,619 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:30,619 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:30,619 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:30,619 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,619 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:30,619 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:30,620 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:30,621 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,621 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,621 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,622 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:30,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fbd73d7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:30,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:30,626 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,629 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:30,632 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:30,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:30,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:30,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:30,632 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:30,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,637 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,638 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:30,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:30,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,643 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:30,643 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:30,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:30,743 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:30,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:30,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:30,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:30,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:30,747 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:30,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:30,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:30,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:30,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:30,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:30,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:30,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:30,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:30,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:30,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:30,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332e529c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:30,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:30,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:30,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:30,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:30,764 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:30,768 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:30,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:30,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:30,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:30,768 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:30,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,774 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,781 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:30,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:30,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,781 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:30,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,781 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:30,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:30,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601330789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601330789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:30,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601330789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,878 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/fa6cc7068a6146a4a408eb5b71190527 2023-03-23 19:54:30,882 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:30,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:30,888 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:30,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:30,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:30,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,890 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:30,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:30,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:30,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:30,893 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,893 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:30,893 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:30,893 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:30,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a647c40[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:30,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:30,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,897 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:30,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:30,897 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:30,898 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:30,900 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,900 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:30,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:30,902 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:30,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:30,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:30,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:30,903 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:30,904 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,904 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:30,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,904 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:30,905 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,906 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:30,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:30,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:30,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:30,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:30,908 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:30,908 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,909 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:30,909 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:30,909 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:30,909 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:30,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,009 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:31,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:31,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:31,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:31,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:31,014 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:31,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:31,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:31,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:31,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:31,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:31,020 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:31,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9000d1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:31,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,023 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:31,023 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,023 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:31,023 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:31,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:31,027 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:31,027 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:31,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:31,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:31,028 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:31,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:31,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:31,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,040 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:31,040 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,041 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:31,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,041 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,041 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,041 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601331093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601331094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601331095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,141 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:31,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:31,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:31,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:31,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:31,148 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:31,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:31,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:31,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:31,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,156 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:31,157 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,157 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:31,157 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:31,157 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:31,157 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:31,159 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,161 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,161 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f37e7bf[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:31,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:31,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,170 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,170 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:31,170 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:31,170 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:31,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:31,172 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:31,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:31,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:31,176 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:31,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:31,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:31,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:31,176 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:31,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,178 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,180 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,183 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,183 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,183 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:31,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:31,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,183 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:31,183 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:31,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,183 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:31,283 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:31,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:31,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:31,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:31,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:31,286 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:31,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:31,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:31,287 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,287 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,289 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,289 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:31,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:31,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:31,290 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,290 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:31,290 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:31,290 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:31,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:31,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/a99f898b97494305ac4d95b32d353556 2023-03-23 19:54:31,292 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,293 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:31,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4cf077[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:31,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:31,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:31,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:31,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:31,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:31,301 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:31,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:31,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:31,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:31,301 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:31,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:31,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:31,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,306 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/aa0855744a6449aab7d93bc6c52038ee 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:31,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:31,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:31,308 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:31,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,309 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:31,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:31,309 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:31,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/fa6cc7068a6146a4a408eb5b71190527 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527 2023-03-23 19:54:31,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527, entries=400, sequenceid=235, filesize=59.4 K 2023-03-23 19:54:31,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/a99f898b97494305ac4d95b32d353556 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556 2023-03-23 19:54:31,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556, entries=150, sequenceid=235, filesize=11.8 K 2023-03-23 19:54:31,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/aa0855744a6449aab7d93bc6c52038ee as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee 2023-03-23 19:54:31,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee, entries=150, sequenceid=235, filesize=11.8 K 2023-03-23 19:54:31,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 924ms, sequenceid=235, compaction requested=true 2023-03-23 19:54:31,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:31,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:31,378 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:54:31,378 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:54:31,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:31,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:31,380 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 190638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:54:31,380 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/A is initiating minor compaction (all files) 2023-03-23 19:54:31,380 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/A in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,381 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b13a2b270ea04366bafaea2b0a256417, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=186.2 K 2023-03-23 19:54:31,381 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70130 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:54:31,381 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/B is initiating minor compaction (all files) 2023-03-23 19:54:31,381 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/B in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,381 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/44afc49dc1cf410781945473f6fb79a8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=68.5 K 2023-03-23 19:54:31,381 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b13a2b270ea04366bafaea2b0a256417, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:31,381 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 44afc49dc1cf410781945473f6fb79a8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:31,382 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4cecbe1d78584185a2c70a0b068bd329, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679601264633 2023-03-23 19:54:31,383 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9a190140993e4b4aaa3fc3da5a958bd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679601264633 2023-03-23 19:54:31,384 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f6c1ad3242f04d72825d55a9837c5d2d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679601265771 2023-03-23 19:54:31,384 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 238de375080a4d60a7ede8e27fcca011, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679601265771 2023-03-23 19:54:31,384 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5ca30040795345ac88b9bd159dbcce44, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679601267910 2023-03-23 19:54:31,385 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 559bfc1cc6f6489e9fc8d829350f0b57, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679601267911 2023-03-23 19:54:31,385 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d60871d5a0be4239baca70ff638158da, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601267987 2023-03-23 19:54:31,385 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ae6961380de642599d1a6e20ea3dd95c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601267987 2023-03-23 19:54:31,385 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fa6cc7068a6146a4a408eb5b71190527, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679601270138 2023-03-23 19:54:31,385 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a99f898b97494305ac4d95b32d353556, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679601270450 2023-03-23 19:54:31,409 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:31,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:31,429 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#B#compaction#171 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:31,454 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:31,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:31,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:31,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:31,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:31,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:31,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:31,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:31,491 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#A#compaction#172 average throughput is 0.42 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:31,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:31,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:31,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:31,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:31,500 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:31,501 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:31,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:31,533 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303231d4feb9f223546bdad220de92a534e72 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231d4feb9f223546bdad220de92a534e72 2023-03-23 19:54:31,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:31,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:31,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601331606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601331607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601331610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601331714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601331714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601331716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,912 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/a44d5848e21e496a9f1a7d33e295d8c7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a44d5848e21e496a9f1a7d33e295d8c7 2023-03-23 19:54:31,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601331917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601331918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601331919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:31,921 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/B of 2301f05dcd8098430ba2e76ffa6a3bb0 into a44d5848e21e496a9f1a7d33e295d8c7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:31,921 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:31,921 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/B, priority=10, startTime=1679601271378; duration=0sec 2023-03-23 19:54:31,921 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:31,922 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:54:31,926 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70130 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:54:31,926 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 2301f05dcd8098430ba2e76ffa6a3bb0/C is initiating minor compaction (all files) 2023-03-23 19:54:31,926 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2301f05dcd8098430ba2e76ffa6a3bb0/C in TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:31,926 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6b4ec2e5d61e46548538dbee5da14b92, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp, totalSize=68.5 K 2023-03-23 19:54:31,927 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6b4ec2e5d61e46548538dbee5da14b92, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679601263479 2023-03-23 19:54:31,930 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4da1bf9d055148759ceae427e3f523b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679601264633 2023-03-23 19:54:31,931 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a92400eb569e423ba60a8bd162a04f0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679601265771 2023-03-23 19:54:31,931 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ab543750546d4f5b8a5444fb20c0b805, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679601267911 2023-03-23 19:54:31,932 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 899ac687e68a4f0ebe5ca14748c70c7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601267987 2023-03-23 19:54:31,932 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting aa0855744a6449aab7d93bc6c52038ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679601270450 2023-03-23 19:54:31,945 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/7ae6dd57ef2545fb8a4726ded0d83d33 2023-03-23 19:54:31,947 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 2301f05dcd8098430ba2e76ffa6a3bb0#C#compaction#174 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:32,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601332220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601332220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601332222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,358 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/20c66f56ec3b4348a382870c0f271b42 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/20c66f56ec3b4348a382870c0f271b42 2023-03-23 19:54:32,363 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/A of 2301f05dcd8098430ba2e76ffa6a3bb0 into 20c66f56ec3b4348a382870c0f271b42(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:32,363 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:32,363 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/A, priority=10, startTime=1679601271377; duration=0sec 2023-03-23 19:54:32,363 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:32,371 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/c1820af0dac74e39859ea37bfe2216a5 2023-03-23 19:54:32,373 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/6043b0a49b9b4d21b0c4f78431e93bdd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6043b0a49b9b4d21b0c4f78431e93bdd 2023-03-23 19:54:32,380 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2301f05dcd8098430ba2e76ffa6a3bb0/C of 2301f05dcd8098430ba2e76ffa6a3bb0 into 6043b0a49b9b4d21b0c4f78431e93bdd(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:32,380 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:32,380 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0., storeName=2301f05dcd8098430ba2e76ffa6a3bb0/C, priority=10, startTime=1679601271378; duration=0sec 2023-03-23 19:54:32,380 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:32,384 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/c7447923b00b4b6e82c25f1cc099a88b 2023-03-23 19:54:32,388 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/7ae6dd57ef2545fb8a4726ded0d83d33 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/7ae6dd57ef2545fb8a4726ded0d83d33 2023-03-23 19:54:32,392 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/7ae6dd57ef2545fb8a4726ded0d83d33, entries=150, sequenceid=255, filesize=25.3 K 2023-03-23 19:54:32,393 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/c1820af0dac74e39859ea37bfe2216a5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/c1820af0dac74e39859ea37bfe2216a5 2023-03-23 19:54:32,399 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/c1820af0dac74e39859ea37bfe2216a5, entries=150, sequenceid=255, filesize=11.8 K 2023-03-23 19:54:32,400 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/c7447923b00b4b6e82c25f1cc099a88b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/c7447923b00b4b6e82c25f1cc099a88b 2023-03-23 19:54:32,404 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/c7447923b00b4b6e82c25f1cc099a88b, entries=150, sequenceid=255, filesize=11.8 K 2023-03-23 19:54:32,405 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 904ms, sequenceid=255, compaction requested=false 2023-03-23 19:54:32,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:32,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:32,408 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,408 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:32,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:32,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f30d82[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:32,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:32,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,414 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:32,416 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:32,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:32,417 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:32,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:32,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:32,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:32,417 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:32,423 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:32,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:32,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:32,426 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,427 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:32,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:32,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:32,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:32,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:32,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:32,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:32,431 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:32,431 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:32,531 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:32,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:32,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:32,535 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:32,535 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:32,535 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:32,535 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:32,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:32,537 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:32,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:32,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,537 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:32,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:32,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:32,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:32,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:32,539 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:32,539 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:32,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:32,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:32,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:32,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601332743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601332743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601332745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601332846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601332846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:32,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601332849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:32,952 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303238b64c93b882148698997285b7f4f87a3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238b64c93b882148698997285b7f4f87a3 2023-03-23 19:54:32,958 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/29af98b0949845679a4a448c47cd2287 2023-03-23 19:54:33,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601333051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601333052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601333052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601333355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601333355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601333355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,367 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/a3c1963fa3d74cfb85375a872bba6257 2023-03-23 19:54:33,378 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/0b61aba258884d8e8f797b48bd870328 2023-03-23 19:54:33,381 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/29af98b0949845679a4a448c47cd2287 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/29af98b0949845679a4a448c47cd2287 2023-03-23 19:54:33,385 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/29af98b0949845679a4a448c47cd2287, entries=150, sequenceid=270, filesize=25.4 K 2023-03-23 19:54:33,386 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/a3c1963fa3d74cfb85375a872bba6257 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a3c1963fa3d74cfb85375a872bba6257 2023-03-23 19:54:33,390 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a3c1963fa3d74cfb85375a872bba6257, entries=150, sequenceid=270, filesize=11.9 K 2023-03-23 19:54:33,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/0b61aba258884d8e8f797b48bd870328 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/0b61aba258884d8e8f797b48bd870328 2023-03-23 19:54:33,395 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/0b61aba258884d8e8f797b48bd870328, entries=150, sequenceid=270, filesize=11.9 K 2023-03-23 19:54:33,396 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 857ms, sequenceid=270, compaction requested=true 2023-03-23 19:54:33,396 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:33,396 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:33,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:33,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:33,400 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:33,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:33,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c9e1325[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:33,402 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:33,402 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:33,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,404 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:33,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:33,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:33,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:33,405 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:33,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:33,410 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:33,410 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:33,410 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:33,410 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:33,410 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:33,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:33,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:33,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:33,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,418 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,421 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,421 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:33,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:33,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:33,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:33,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:33,422 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:33,422 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:33,422 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:33,522 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:33,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:33,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:33,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:33,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:33,526 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:33,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:33,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:33,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:33,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:33,528 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:33,528 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:33,528 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,528 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:33,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:33,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:33,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:33,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:33,530 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:33,530 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. started... 2023-03-23 19:54:33,530 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:33,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:33,554 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032368c94bd35f9b407c9589191221687de3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368c94bd35f9b407c9589191221687de3 2023-03-23 19:54:33,579 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/06b455d163054594a5f61efd51acb437 2023-03-23 19:54:33,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:33,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. as already flushing 2023-03-23 19:54:33,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601333869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601333869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601333870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601333970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601333972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:33,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601333972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:33,976 DEBUG [Thread-969] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:60301 2023-03-23 19:54:33,976 DEBUG [Thread-969] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:33,977 DEBUG [Thread-967] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:60301 2023-03-23 19:54:33,977 DEBUG [Thread-967] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:33,978 DEBUG [Thread-971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:60301 2023-03-23 19:54:33,978 DEBUG [Thread-971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:33,980 DEBUG [Thread-973] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:60301 2023-03-23 19:54:33,980 DEBUG [Thread-973] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:33,990 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/b48b4bb7984648f983940bc7a3d06e7d 2023-03-23 19:54:34,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:34,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48552 deadline: 1679601334172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:34,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:34,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48532 deadline: 1679601334174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48516 deadline: 1679601334174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,400 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/7545284c71c84b73aa41d3de344f1d49 2023-03-23 19:54:34,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/06b455d163054594a5f61efd51acb437 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/06b455d163054594a5f61efd51acb437 2023-03-23 19:54:34,409 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/06b455d163054594a5f61efd51acb437, entries=150, sequenceid=290, filesize=25.4 K 2023-03-23 19:54:34,410 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/b48b4bb7984648f983940bc7a3d06e7d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b48b4bb7984648f983940bc7a3d06e7d 2023-03-23 19:54:34,414 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b48b4bb7984648f983940bc7a3d06e7d, entries=150, sequenceid=290, filesize=11.9 K 2023-03-23 19:54:34,415 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/7545284c71c84b73aa41d3de344f1d49 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7545284c71c84b73aa41d3de344f1d49 2023-03-23 19:54:34,419 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7545284c71c84b73aa41d3de344f1d49, entries=150, sequenceid=290, filesize=11.9 K 2023-03-23 19:54:34,419 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 889ms, sequenceid=290, compaction requested=true 2023-03-23 19:54:34,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:34,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:34,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:34,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:34,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:34,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:34,421 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:34,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd41916[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:34,423 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:34,423 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,424 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:34,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:34,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:34,426 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:34,428 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:34,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:34,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:34,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:34,428 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:34,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:34,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:34,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:34,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:34,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:34,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:34,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:34,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:34,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:34,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:34,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:34,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:34,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:34,434 DEBUG [Thread-966] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:34,434 DEBUG [Thread-966] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:34,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:34,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:34,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:34,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:54:34,476 DEBUG [Thread-960] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:60301 2023-03-23 19:54:34,476 DEBUG [Thread-962] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:60301 2023-03-23 19:54:34,476 DEBUG [Thread-960] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:34,476 DEBUG [Thread-962] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:34,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:34,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:34,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:34,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:34,479 DEBUG [Thread-964] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:60301 2023-03-23 19:54:34,479 DEBUG [Thread-964] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,534 DEBUG [Thread-966] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:34,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:34,574 DEBUG [Thread-958] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:60301 2023-03-23 19:54:34,574 DEBUG [Thread-958] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,583 DEBUG [Thread-956] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:60301 2023-03-23 19:54:34,583 DEBUG [Thread-956] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-03-23 19:54:34,583 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5103 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5198 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2152 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6455 rows 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2125 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6375 rows 2023-03-23 19:54:34,584 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:54:34,584 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:60301 2023-03-23 19:54:34,584 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:54:34,586 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:54:34,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:54:34,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:34,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:34,592 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601274591"}]},"ts":"1679601274591"} 2023-03-23 19:54:34,593 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:54:34,595 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:54:34,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, UNASSIGN}] 2023-03-23 19:54:34,597 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, UNASSIGN 2023-03-23 19:54:34,598 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:34,598 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601274598"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601274598"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601274598"}]},"ts":"1679601274598"} 2023-03-23 19:54:34,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:34,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:34,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2301f05dcd8098430ba2e76ffa6a3bb0, disabling compactions & flushes 2023-03-23 19:54:34,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:34,892 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303231c751b77535248af8cb120bda1d99952 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231c751b77535248af8cb120bda1d99952 2023-03-23 19:54:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:35,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:35,297 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/b037e2b51be543349567ff16291baf7b 2023-03-23 19:54:35,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:35,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/400dc2ca8e71466885f5a69a6e5f8736 2023-03-23 19:54:36,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/39df3ea0062a442e823b29528ea5eb27 2023-03-23 19:54:36,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/b037e2b51be543349567ff16291baf7b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b037e2b51be543349567ff16291baf7b 2023-03-23 19:54:36,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b037e2b51be543349567ff16291baf7b, entries=150, sequenceid=304, filesize=25.4 K 2023-03-23 19:54:36,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/400dc2ca8e71466885f5a69a6e5f8736 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/400dc2ca8e71466885f5a69a6e5f8736 2023-03-23 19:54:36,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/400dc2ca8e71466885f5a69a6e5f8736, entries=150, sequenceid=304, filesize=11.9 K 2023-03-23 19:54:36,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/39df3ea0062a442e823b29528ea5eb27 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/39df3ea0062a442e823b29528ea5eb27 2023-03-23 19:54:36,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/39df3ea0062a442e823b29528ea5eb27, entries=150, sequenceid=304, filesize=11.9 K 2023-03-23 19:54:36,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1655ms, sequenceid=304, compaction requested=true 2023-03-23 19:54:36,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:36,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:36,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:36,132 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. because compaction request was cancelled 2023-03-23 19:54:36,132 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. because compaction request was cancelled 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:36,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. after waiting 0 ms 2023-03-23 19:54:36,132 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. because compaction request was cancelled 2023-03-23 19:54:36,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:36,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2301f05dcd8098430ba2e76ffa6a3bb0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=A 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:36,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=B 2023-03-23 19:54:36,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:36,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2301f05dcd8098430ba2e76ffa6a3bb0, store=C 2023-03-23 19:54:36,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:36,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323734e26a0dd894c9e8bb3c22d4b620582 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323734e26a0dd894c9e8bb3c22d4b620582 2023-03-23 19:54:36,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:36,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/9cba03b7e3e640f59cb0b2185e110caa 2023-03-23 19:54:37,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e9984765930a4a9fa1fd47150e643d4e 2023-03-23 19:54:37,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/17772fc466384329a3399d4f39152de2 2023-03-23 19:54:37,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/A/9cba03b7e3e640f59cb0b2185e110caa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/9cba03b7e3e640f59cb0b2185e110caa 2023-03-23 19:54:37,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/9cba03b7e3e640f59cb0b2185e110caa, entries=150, sequenceid=310, filesize=25.4 K 2023-03-23 19:54:37,775 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/B/e9984765930a4a9fa1fd47150e643d4e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e9984765930a4a9fa1fd47150e643d4e 2023-03-23 19:54:37,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e9984765930a4a9fa1fd47150e643d4e, entries=150, sequenceid=310, filesize=11.9 K 2023-03-23 19:54:37,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/.tmp/C/17772fc466384329a3399d4f39152de2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/17772fc466384329a3399d4f39152de2 2023-03-23 19:54:37,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/17772fc466384329a3399d4f39152de2, entries=150, sequenceid=310, filesize=11.9 K 2023-03-23 19:54:37,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2301f05dcd8098430ba2e76ffa6a3bb0 in 1651ms, sequenceid=310, compaction requested=true 2023-03-23 19:54:37,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b13a2b270ea04366bafaea2b0a256417, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527] to archive 2023-03-23 19:54:37,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:37,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/eae3ff906f764758bcb0e07d9b55bbeb 2023-03-23 19:54:37,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/78c4c04d899f44939d98913af7e5431f 2023-03-23 19:54:37,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/991351fd214b40948323f8d72e0a851d 2023-03-23 19:54:37,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b01e381e55bd4fd0914af0c94c2c4298 2023-03-23 19:54:37,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/1f13bcec85114cb4866b726fd7793dd5 2023-03-23 19:54:37,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/195da42c3e5d4227877503ec27a4f463 2023-03-23 19:54:37,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0e6843130466449ba0a3cc0e60103dc9 2023-03-23 19:54:37,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/0be0f190f1d84325bc6a29fb72dcacbd 2023-03-23 19:54:37,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/adcf21b1d915459a838351eb21860d77 2023-03-23 19:54:37,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b13a2b270ea04366bafaea2b0a256417 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b13a2b270ea04366bafaea2b0a256417 2023-03-23 19:54:37,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/4cecbe1d78584185a2c70a0b068bd329 2023-03-23 19:54:37,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/f6c1ad3242f04d72825d55a9837c5d2d 2023-03-23 19:54:37,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/5ca30040795345ac88b9bd159dbcce44 2023-03-23 19:54:37,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/d60871d5a0be4239baca70ff638158da 2023-03-23 19:54:37,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/fa6cc7068a6146a4a408eb5b71190527 2023-03-23 19:54:37,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/44afc49dc1cf410781945473f6fb79a8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556] to archive 2023-03-23 19:54:37,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:37,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/49658aed4ba247f6ac2f8139fb70b0bd 2023-03-23 19:54:37,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b1e2832d566c40339b57f93db29c7da3 2023-03-23 19:54:37,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3ae2326465de4cc8b6c06ba72227e653 2023-03-23 19:54:37,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/5898cbf146dd4a2e8fa8d0218ebdc1a3 2023-03-23 19:54:37,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/3eaa188d9e3b4832a7d42cb3afd1d78b 2023-03-23 19:54:37,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e1af1babfdb54df4a291deadac33ed82 2023-03-23 19:54:37,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e5c5016d51df44ba9019d8cdfc7b611b 2023-03-23 19:54:37,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/90fd6eaa6ef3476aa074c47d8595bf1b 2023-03-23 19:54:37,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/44afc49dc1cf410781945473f6fb79a8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/44afc49dc1cf410781945473f6fb79a8 2023-03-23 19:54:37,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/0d3b9aebbda4479292566926dd86ece0 2023-03-23 19:54:37,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/9a190140993e4b4aaa3fc3da5a958bd0 2023-03-23 19:54:37,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/238de375080a4d60a7ede8e27fcca011 2023-03-23 19:54:37,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/559bfc1cc6f6489e9fc8d829350f0b57 2023-03-23 19:54:37,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/ae6961380de642599d1a6e20ea3dd95c 2023-03-23 19:54:37,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a99f898b97494305ac4d95b32d353556 2023-03-23 19:54:37,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6b4ec2e5d61e46548538dbee5da14b92, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee] to archive 2023-03-23 19:54:37,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:54:37,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/2c9b43cf284e43b89485f0ba629b880a 2023-03-23 19:54:37,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ddb7e6edb98348c286e00a1178d8cdae 2023-03-23 19:54:37,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/5c095fe6ecec412aa824d9f7cc2c06e9 2023-03-23 19:54:37,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/f2ce91f4b8124e79b148dc998dbb5e48 2023-03-23 19:54:37,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/e9702b9df4584e4681f57fc85f78a3f7 2023-03-23 19:54:37,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/53dab775e2ab4f4ea8da3ddc8d893f9a 2023-03-23 19:54:37,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7941e0d32f1040dca7f0dede170c7f21 2023-03-23 19:54:37,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/86a2ca598ebf4932b8c233733c4b7e0f 2023-03-23 19:54:37,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6b4ec2e5d61e46548538dbee5da14b92 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6b4ec2e5d61e46548538dbee5da14b92 2023-03-23 19:54:37,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ff7302baeea44ab1b37ce3c99ce7d89b 2023-03-23 19:54:37,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/4da1bf9d055148759ceae427e3f523b1 2023-03-23 19:54:37,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/a92400eb569e423ba60a8bd162a04f0c 2023-03-23 19:54:37,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/ab543750546d4f5b8a5444fb20c0b805 2023-03-23 19:54:37,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/899ac687e68a4f0ebe5ca14748c70c7b 2023-03-23 19:54:37,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/aa0855744a6449aab7d93bc6c52038ee 2023-03-23 19:54:37,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits/313.seqid, newMaxSeqId=313, maxSeqId=4 2023-03-23 19:54:37,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0. 2023-03-23 19:54:37,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2301f05dcd8098430ba2e76ffa6a3bb0: 2023-03-23 19:54:37,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:37,866 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2301f05dcd8098430ba2e76ffa6a3bb0, regionState=CLOSED 2023-03-23 19:54:37,866 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601277866"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601277866"}]},"ts":"1679601277866"} 2023-03-23 19:54:37,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-23 19:54:37,869 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 2301f05dcd8098430ba2e76ffa6a3bb0, server=jenkins-hbase4.apache.org,34277,1679601218015 in 3.2680 sec 2023-03-23 19:54:37,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-23 19:54:37,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2301f05dcd8098430ba2e76ffa6a3bb0, UNASSIGN in 3.2720 sec 2023-03-23 19:54:37,870 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601277870"}]},"ts":"1679601277870"} 2023-03-23 19:54:37,871 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:54:37,872 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:54:37,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.2870 sec 2023-03-23 19:54:38,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-23 19:54:38,697 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-23 19:54:38,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:54:38,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,700 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-23 19:54:38,700 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,705 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:38,707 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits] 2023-03-23 19:54:38,710 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/06b455d163054594a5f61efd51acb437 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/06b455d163054594a5f61efd51acb437 2023-03-23 19:54:38,711 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/20c66f56ec3b4348a382870c0f271b42 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/20c66f56ec3b4348a382870c0f271b42 2023-03-23 19:54:38,712 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/29af98b0949845679a4a448c47cd2287 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/29af98b0949845679a4a448c47cd2287 2023-03-23 19:54:38,713 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/7ae6dd57ef2545fb8a4726ded0d83d33 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/7ae6dd57ef2545fb8a4726ded0d83d33 2023-03-23 19:54:38,714 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/9cba03b7e3e640f59cb0b2185e110caa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/9cba03b7e3e640f59cb0b2185e110caa 2023-03-23 19:54:38,716 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b037e2b51be543349567ff16291baf7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/A/b037e2b51be543349567ff16291baf7b 2023-03-23 19:54:38,718 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/400dc2ca8e71466885f5a69a6e5f8736 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/400dc2ca8e71466885f5a69a6e5f8736 2023-03-23 19:54:38,719 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a3c1963fa3d74cfb85375a872bba6257 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a3c1963fa3d74cfb85375a872bba6257 2023-03-23 19:54:38,720 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a44d5848e21e496a9f1a7d33e295d8c7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/a44d5848e21e496a9f1a7d33e295d8c7 2023-03-23 19:54:38,721 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b48b4bb7984648f983940bc7a3d06e7d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/b48b4bb7984648f983940bc7a3d06e7d 2023-03-23 19:54:38,722 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/c1820af0dac74e39859ea37bfe2216a5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/c1820af0dac74e39859ea37bfe2216a5 2023-03-23 19:54:38,723 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e9984765930a4a9fa1fd47150e643d4e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/B/e9984765930a4a9fa1fd47150e643d4e 2023-03-23 19:54:38,725 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/0b61aba258884d8e8f797b48bd870328 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/0b61aba258884d8e8f797b48bd870328 2023-03-23 19:54:38,726 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/17772fc466384329a3399d4f39152de2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/17772fc466384329a3399d4f39152de2 2023-03-23 19:54:38,727 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/39df3ea0062a442e823b29528ea5eb27 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/39df3ea0062a442e823b29528ea5eb27 2023-03-23 19:54:38,728 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6043b0a49b9b4d21b0c4f78431e93bdd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/6043b0a49b9b4d21b0c4f78431e93bdd 2023-03-23 19:54:38,729 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7545284c71c84b73aa41d3de344f1d49 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/7545284c71c84b73aa41d3de344f1d49 2023-03-23 19:54:38,730 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/c7447923b00b4b6e82c25f1cc099a88b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/C/c7447923b00b4b6e82c25f1cc099a88b 2023-03-23 19:54:38,734 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits/313.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0/recovered.edits/313.seqid 2023-03-23 19:54:38,736 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/2301f05dcd8098430ba2e76ffa6a3bb0 2023-03-23 19:54:38,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:54:38,736 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:54:38,737 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-23 19:54:38,741 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323034d977cebd44f27bcf32a17759cbaca to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323034d977cebd44f27bcf32a17759cbaca 2023-03-23 19:54:38,743 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231ab22e54879d443598a3a571813d0ad4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231ab22e54879d443598a3a571813d0ad4 2023-03-23 19:54:38,744 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231c751b77535248af8cb120bda1d99952 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231c751b77535248af8cb120bda1d99952 2023-03-23 19:54:38,745 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231d4feb9f223546bdad220de92a534e72 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303231d4feb9f223546bdad220de92a534e72 2023-03-23 19:54:38,746 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234e9428196fd54bd8abec99726022422c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234e9428196fd54bd8abec99726022422c 2023-03-23 19:54:38,748 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368c94bd35f9b407c9589191221687de3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368c94bd35f9b407c9589191221687de3 2023-03-23 19:54:38,749 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032371cbc30b1f044a4d8afbdaffb8251dc5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032371cbc30b1f044a4d8afbdaffb8251dc5 2023-03-23 19:54:38,750 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323734e26a0dd894c9e8bb3c22d4b620582 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323734e26a0dd894c9e8bb3c22d4b620582 2023-03-23 19:54:38,751 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032385581c3c691a49cb9137e397cb160f47 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032385581c3c691a49cb9137e397cb160f47 2023-03-23 19:54:38,753 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238a601cd4b8d74dc7bd3927c8699d890a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238a601cd4b8d74dc7bd3927c8699d890a 2023-03-23 19:54:38,754 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238b64c93b882148698997285b7f4f87a3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303238b64c93b882148698997285b7f4f87a3 2023-03-23 19:54:38,755 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323947d776ec6b84f0bb83b53850e71df5f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323947d776ec6b84f0bb83b53850e71df5f 2023-03-23 19:54:38,756 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239e1cf59a819a4b378463ac4f752df307 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239e1cf59a819a4b378463ac4f752df307 2023-03-23 19:54:38,757 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f46ebd07f22489fa3d8914579158266 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f46ebd07f22489fa3d8914579158266 2023-03-23 19:54:38,758 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a241a94b17a84c268e1ea34a06920e00 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a241a94b17a84c268e1ea34a06920e00 2023-03-23 19:54:38,759 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323acff883c0f39404fa2ec2d1bfacc315f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323acff883c0f39404fa2ec2d1bfacc315f 2023-03-23 19:54:38,760 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d165db2efba24b9e86bfdbd52db3edb6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d165db2efba24b9e86bfdbd52db3edb6 2023-03-23 19:54:38,761 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb31651ca4ea48f2b1a6a197c66f420a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb31651ca4ea48f2b1a6a197c66f420a 2023-03-23 19:54:38,762 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323efbcaad7caa24852ab7a9ee31986bd7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323efbcaad7caa24852ab7a9ee31986bd7b 2023-03-23 19:54:38,763 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:54:38,765 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,766 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:54:38,768 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:54:38,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:54:38,769 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601278769"}]},"ts":"9223372036854775807"} 2023-03-23 19:54:38,771 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:54:38,771 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2301f05dcd8098430ba2e76ffa6a3bb0, NAME => 'TestAcidGuarantees,,1679601250836.2301f05dcd8098430ba2e76ffa6a3bb0.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:54:38,771 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:54:38,771 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601278771"}]},"ts":"9223372036854775807"} 2023-03-23 19:54:38,772 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:54:38,774 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 77 msec 2023-03-23 19:54:38,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-23 19:54:38,801 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-23 19:54:38,813 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 232) - Thread LEAK? -, OpenFileDescriptor=542 (was 578), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=377 (was 358) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5466 (was 5629) 2023-03-23 19:54:38,823 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=542, MaxFileDescriptor=60000, SystemLoadAverage=377, ProcessCount=170, AvailableMemoryMB=5466 2023-03-23 19:54:38,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:54:38,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:54:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:54:38,827 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:54:38,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-23 19:54:38,828 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:54:38,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:38,829 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:38,830 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa empty. 2023-03-23 19:54:38,830 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:38,830 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:54:38,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:39,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:39,238 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:54:39,240 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 18eab3ab7700f84be1d9a1a02e88ccfa, NAME => 'TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:54:39,296 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-23 19:54:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:39,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:54:39,645 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 18eab3ab7700f84be1d9a1a02e88ccfa, disabling compactions & flushes 2023-03-23 19:54:39,645 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,646 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,646 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. after waiting 0 ms 2023-03-23 19:54:39,646 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,646 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,646 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:39,648 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:54:39,649 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601279648"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601279648"}]},"ts":"1679601279648"} 2023-03-23 19:54:39,650 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:54:39,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:54:39,650 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601279650"}]},"ts":"1679601279650"} 2023-03-23 19:54:39,651 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:54:39,655 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, ASSIGN}] 2023-03-23 19:54:39,656 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, ASSIGN 2023-03-23 19:54:39,657 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:54:39,808 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=18eab3ab7700f84be1d9a1a02e88ccfa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:39,808 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601279808"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601279808"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601279808"}]},"ts":"1679601279808"} 2023-03-23 19:54:39,809 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:54:39,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:39,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 18eab3ab7700f84be1d9a1a02e88ccfa, NAME => 'TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:54:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:54:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,968 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,969 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A 2023-03-23 19:54:39,969 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A 2023-03-23 19:54:39,969 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:39,969 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 18eab3ab7700f84be1d9a1a02e88ccfa columnFamilyName A 2023-03-23 19:54:39,970 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(310): Store=18eab3ab7700f84be1d9a1a02e88ccfa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:39,970 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,971 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B 2023-03-23 19:54:39,971 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B 2023-03-23 19:54:39,971 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:39,972 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 18eab3ab7700f84be1d9a1a02e88ccfa columnFamilyName B 2023-03-23 19:54:39,972 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(310): Store=18eab3ab7700f84be1d9a1a02e88ccfa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:39,972 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,973 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C 2023-03-23 19:54:39,973 DEBUG [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C 2023-03-23 19:54:39,973 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:54:39,973 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 18eab3ab7700f84be1d9a1a02e88ccfa columnFamilyName C 2023-03-23 19:54:39,974 INFO [StoreOpener-18eab3ab7700f84be1d9a1a02e88ccfa-1] regionserver.HStore(310): Store=18eab3ab7700f84be1d9a1a02e88ccfa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:54:39,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:54:39,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:39,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:54:39,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 18eab3ab7700f84be1d9a1a02e88ccfa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71065984, jitterRate=0.05896568298339844}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:54:39,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:39,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., pid=30, masterSystemTime=1679601279961 2023-03-23 19:54:39,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:39,984 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=18eab3ab7700f84be1d9a1a02e88ccfa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:39,984 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601279983"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601279983"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601279983"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601279983"}]},"ts":"1679601279983"} 2023-03-23 19:54:39,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-23 19:54:39,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 176 msec 2023-03-23 19:54:39,987 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-23 19:54:39,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, ASSIGN in 331 msec 2023-03-23 19:54:39,988 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:54:39,988 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601279988"}]},"ts":"1679601279988"} 2023-03-23 19:54:39,989 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:54:39,991 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:54:39,992 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1660 sec 2023-03-23 19:54:40,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-23 19:54:40,933 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-23 19:54:40,933 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,944 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,945 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:40,946 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46448, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:40,947 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:54:40,949 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:54:40,950 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,954 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,954 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,957 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,958 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,961 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,961 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,969 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,969 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,975 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,975 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,980 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,980 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,986 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,986 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,991 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,991 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,995 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:40,996 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:54:40,999 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:54:41,003 DEBUG [hconnection-0x4cc54edf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,003 DEBUG [hconnection-0x48d58389-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,003 DEBUG [hconnection-0x3f4a8b1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,003 DEBUG [hconnection-0x13e289a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,004 DEBUG [hconnection-0x307580f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,004 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,004 DEBUG [hconnection-0x1bc0ce4a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,005 DEBUG [hconnection-0x2c58e509-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,006 DEBUG [hconnection-0x65e75af5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,007 DEBUG [hconnection-0x63a25938-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,007 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46504, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,009 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,009 DEBUG [hconnection-0x82a470c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:54:41,009 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46488, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,009 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,009 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,012 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46510, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,013 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46482, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,018 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46506, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,019 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:54:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,019 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,019 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,019 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,019 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,025 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,031 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,031 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,032 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:41,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601281052, max timestamp=1679601281059Num uniques 859; 2023-03-23 19:54:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601281052, max timestamp=1679601281059Num uniques 859; 2023-03-23 19:54:41,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:41,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601281052, max timestamp=1679601281059Num uniques 859; 2023-03-23 19:54:41,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:41,077 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/7cc4bb4797304b82a1012ead5cda8e4b 2023-03-23 19:54:41,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601341085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601341085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601341085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601281082, max timestamp=1679601281087Num uniques 1068; 2023-03-23 19:54:41,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601281082, max timestamp=1679601281087Num uniques 1068; 2023-03-23 19:54:41,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-23 19:54:41,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-23 19:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601281082, max timestamp=1679601281087Num uniques 1068; 2023-03-23 19:54:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-23 19:54:41,102 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/6a8a2ab33763459eb7e2cfa955ee77fe 2023-03-23 19:54:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601281121, max timestamp=1679601281127Num uniques 1027; 2023-03-23 19:54:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-23 19:54:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601281121, max timestamp=1679601281127Num uniques 1027; 2023-03-23 19:54:41,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-23 19:54:41,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601281121, max timestamp=1679601281127Num uniques 1027; 2023-03-23 19:54:41,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-23 19:54:41,134 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9c13183b938f49819e6d3c7725ea533a 2023-03-23 19:54:41,139 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/7cc4bb4797304b82a1012ead5cda8e4b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b 2023-03-23 19:54:41,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b, entries=150, sequenceid=9, filesize=11.6 K 2023-03-23 19:54:41,146 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/6a8a2ab33763459eb7e2cfa955ee77fe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe 2023-03-23 19:54:41,158 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe, entries=150, sequenceid=9, filesize=11.6 K 2023-03-23 19:54:41,159 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9c13183b938f49819e6d3c7725ea533a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a 2023-03-23 19:54:41,166 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a, entries=150, sequenceid=9, filesize=11.6 K 2023-03-23 19:54:41,168 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=93.93 KB/96180 for 18eab3ab7700f84be1d9a1a02e88ccfa in 136ms, sequenceid=9, compaction requested=false 2023-03-23 19:54:41,168 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-23 19:54:41,170 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:41,170 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:41,172 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=107.34 KB heapSize=303.61 KB 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:41,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:41,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654289b3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,177 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,178 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,182 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,183 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,195 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,196 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,196 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:41,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601341192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601341193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601341194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601341196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601341197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4fe85194410d47629e30d2f9ed2f8417 2023-03-23 19:54:41,296 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,300 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601341298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,302 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601341301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,304 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,304 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,304 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,304 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601341302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a421784[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,310 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,314 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,314 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,314 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,314 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,314 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,316 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,316 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,316 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,321 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,321 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601341398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601341399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,421 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,425 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,428 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,430 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,430 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,430 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,430 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,432 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764924d2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,437 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,440 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,440 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,443 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,450 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,450 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,451 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,451 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601341502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601341505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601341505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,551 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,556 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,558 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,558 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,558 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,558 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a499fa4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,564 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,564 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,564 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,565 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,566 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,568 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,568 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,570 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,576 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,576 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:41,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/2f76c9b19b3344edb009e491dffe535e 2023-03-23 19:54:41,676 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c7fbc02d72394e02b7efd1a447273484 2023-03-23 19:54:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,680 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4fe85194410d47629e30d2f9ed2f8417 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417 2023-03-23 19:54:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417, entries=200, sequenceid=76, filesize=13.9 K 2023-03-23 19:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/2f76c9b19b3344edb009e491dffe535e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e 2023-03-23 19:54:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bb78c4f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e, entries=150, sequenceid=76, filesize=11.6 K 2023-03-23 19:54:41,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c7fbc02d72394e02b7efd1a447273484 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484 2023-03-23 19:54:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,697 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,697 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484, entries=150, sequenceid=76, filesize=11.6 K 2023-03-23 19:54:41,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~303.56 KB/310848, currentSize=73.80 KB/75570 for 18eab3ab7700f84be1d9a1a02e88ccfa in 526ms, sequenceid=76, compaction requested=false 2023-03-23 19:54:41,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:41,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,703 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:41,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-23 19:54:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:41,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:41,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:41,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,709 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,709 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:41,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/1b44852599434d578d756597f62fc4d4 2023-03-23 19:54:41,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/ff4f38f890f14c73939191dd73b2a822 2023-03-23 19:54:41,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601341732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601341733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601341805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,809 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601341809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601341810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,816 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,818 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,831 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7123568a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,832 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,834 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,835 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601341838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:41,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601341838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,844 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,845 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,845 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,846 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,847 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,852 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:41,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,852 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,852 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,952 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:41,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:41,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:41,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:41,955 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:41,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:41,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:41,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,958 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,959 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,959 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:41,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:41,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:41,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,960 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:41,961 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:41,961 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:41,961 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:41,961 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:41,961 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,961 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:41,961 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:41,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:41,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7393c8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:41,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:41,969 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,969 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:41,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:41,971 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:41,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:41,973 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:41,974 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:41,974 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:41,974 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:41,974 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:41,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:41,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,975 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:41,976 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:41,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:41,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:41,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:41,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:41,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:41,979 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:41,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:41,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:41,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:41,980 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:41,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,980 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:41,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:41,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601342041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601342042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,080 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:42,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:42,083 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:42,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:42,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:42,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,085 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,085 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,085 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:42,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:42,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:42,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:42,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:42,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:42,089 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,090 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:42,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:42,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e1811d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:42,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,092 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:42,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,093 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:42,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:42,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:42,097 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:42,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:42,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:42,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:42,097 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:42,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,099 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:42,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,103 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:42,103 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:42,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:42,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a34787e59cd847c8896fd6f7c2faeb18 2023-03-23 19:54:42,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/1b44852599434d578d756597f62fc4d4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4 2023-03-23 19:54:42,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4, entries=200, sequenceid=92, filesize=13.9 K 2023-03-23 19:54:42,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/ff4f38f890f14c73939191dd73b2a822 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822 2023-03-23 19:54:42,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822, entries=150, sequenceid=92, filesize=11.6 K 2023-03-23 19:54:42,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a34787e59cd847c8896fd6f7c2faeb18 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18 2023-03-23 19:54:42,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18, entries=150, sequenceid=92, filesize=11.6 K 2023-03-23 19:54:42,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 474ms, sequenceid=92, compaction requested=true 2023-03-23 19:54:42,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:42,199 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:42,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:42,199 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:42,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:42,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:42,200 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:42,200 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:42,200 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,200 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=39.4 K 2023-03-23 19:54:42,201 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:42,201 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:42,201 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,201 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=34.8 K 2023-03-23 19:54:42,202 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7cc4bb4797304b82a1012ead5cda8e4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679601281018 2023-03-23 19:54:42,202 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6a8a2ab33763459eb7e2cfa955ee77fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679601281018 2023-03-23 19:54:42,202 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4fe85194410d47629e30d2f9ed2f8417, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679601281155 2023-03-23 19:54:42,202 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2f76c9b19b3344edb009e491dffe535e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679601281158 2023-03-23 19:54:42,203 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ff4f38f890f14c73939191dd73b2a822, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:42,203 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 1b44852599434d578d756597f62fc4d4, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:42,203 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:42,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:42,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:42,211 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:42,211 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:42,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:42,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,214 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:42,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:42,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:42,220 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:42,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:42,236 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:42,236 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,236 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:42,249 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:42,249 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:42,249 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,249 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:42,249 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:42,250 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,265 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/6c818906d8ce4beba73b2818c1f1bc84 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6c818906d8ce4beba73b2818c1f1bc84 2023-03-23 19:54:42,272 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 6c818906d8ce4beba73b2818c1f1bc84(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:42,272 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:42,272 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601282199; duration=0sec 2023-03-23 19:54:42,272 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:42,273 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:42,274 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:42,274 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:42,274 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,274 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=34.8 K 2023-03-23 19:54:42,275 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9c13183b938f49819e6d3c7725ea533a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679601281018 2023-03-23 19:54:42,276 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c7fbc02d72394e02b7efd1a447273484, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679601281158 2023-03-23 19:54:42,276 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a34787e59cd847c8896fd6f7c2faeb18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:42,282 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/42128d080f0d4825ac1f0781effb0246 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/42128d080f0d4825ac1f0781effb0246 2023-03-23 19:54:42,287 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 42128d080f0d4825ac1f0781effb0246(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:42,287 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:42,288 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601282199; duration=0sec 2023-03-23 19:54:42,288 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:42,290 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:42,292 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/601245484906478990c1f26c78dddaed 2023-03-23 19:54:42,300 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/657a77d45f4b49be9ca350655a98fe8b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/657a77d45f4b49be9ca350655a98fe8b 2023-03-23 19:54:42,306 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 657a77d45f4b49be9ca350655a98fe8b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:42,306 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:42,306 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601282199; duration=0sec 2023-03-23 19:54:42,306 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:42,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:42,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:42,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601342328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601342329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601342330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601342345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601342346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601342431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601342432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601342434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601342633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601342635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601342637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,722 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/9d7266bc46e541c6a87d8e1823cded28 2023-03-23 19:54:42,740 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/7d817cfa025b403a93f064698c434ebd 2023-03-23 19:54:42,745 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/601245484906478990c1f26c78dddaed as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed 2023-03-23 19:54:42,749 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed, entries=150, sequenceid=108, filesize=11.6 K 2023-03-23 19:54:42,750 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/9d7266bc46e541c6a87d8e1823cded28 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28 2023-03-23 19:54:42,755 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28, entries=150, sequenceid=108, filesize=11.6 K 2023-03-23 19:54:42,756 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/7d817cfa025b403a93f064698c434ebd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd 2023-03-23 19:54:42,759 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd, entries=150, sequenceid=108, filesize=11.6 K 2023-03-23 19:54:42,760 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 511ms, sequenceid=108, compaction requested=false 2023-03-23 19:54:42,760 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:42,761 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:42,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:42,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:42,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:42,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,771 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:42,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f13bdba[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:42,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:42,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,775 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:42,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:42,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:42,775 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,775 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:42,777 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:42,777 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,777 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,777 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:42,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:42,779 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:42,779 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:42,779 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:42,780 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:42,780 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:42,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,781 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,781 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,783 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,785 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,786 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,786 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,786 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:42,786 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:42,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:42,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:42,849 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:42,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601342865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601342868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,886 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:42,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:42,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:42,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:42,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:42,896 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:42,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:42,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:42,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,899 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,899 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,899 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:42,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:42,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:42,901 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,901 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:42,901 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:42,901 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:42,901 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:42,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:42,905 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:42,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246f35d4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:42,905 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,909 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:42,909 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,909 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:42,914 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:42,918 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:42,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:42,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:42,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:42,918 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:42,920 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,920 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,920 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:42,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:42,922 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:42,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:42,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:42,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:42,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:42,927 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:42,927 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:42,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:42,928 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:42,928 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:42,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:42,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:42,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601342938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601342939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601342939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601342969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:42,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601342970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,027 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,031 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,033 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,033 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:43,036 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:43,036 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:43,039 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d00f42[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:43,041 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:43,041 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:43,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:43,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,046 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:43,055 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:43,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:43,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:43,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:43,055 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:43,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,077 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:43,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,078 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:43,078 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601343173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601343174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,178 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,181 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,183 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:43,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:43,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:43,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:43,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,188 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,188 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,188 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12dd81ec[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:43,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:43,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:43,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:43,193 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:43,196 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:43,197 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:43,197 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:43,197 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:43,197 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:43,198 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,198 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,202 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,204 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:43,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:43,205 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:43,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,205 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:43,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/960af9037afd440581f5cbe382f08d7b 2023-03-23 19:54:43,305 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,309 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,311 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,313 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,313 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,314 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:43,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40710060[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:43,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:43,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,323 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:43,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:43,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:43,333 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:43,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:43,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:43,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:43,333 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:43,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:43,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:43,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,341 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:43,341 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:43,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:43,395 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:54:43,441 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601343442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,445 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601343444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601343444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,447 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:43,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:43,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:43,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ded4c2f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:43,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:43,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:43,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:43,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:43,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:43,458 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:43,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:43,462 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:43,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:43,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:43,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:43,462 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,468 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:43,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:43,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,472 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:43,472 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:43,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:43,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601343475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:43,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601343477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,573 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,576 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,579 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,581 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,581 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,581 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:43,581 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:43,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:43,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:43,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:43,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:43,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb1eacf[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:43,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:43,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:43,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:43,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:43,592 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:43,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:43,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:43,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:43,592 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:43,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:43,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:43,596 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,600 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:43,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:43,601 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:43,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:43,601 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:43,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/c9d34a409c5d453288f4f46c480caa10 2023-03-23 19:54:43,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c84b22c5511242a1913d8a8c56211327 2023-03-23 19:54:43,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/960af9037afd440581f5cbe382f08d7b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b 2023-03-23 19:54:43,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b, entries=200, sequenceid=128, filesize=13.9 K 2023-03-23 19:54:43,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/c9d34a409c5d453288f4f46c480caa10 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10 2023-03-23 19:54:43,701 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:43,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:43,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:43,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10, entries=150, sequenceid=128, filesize=11.6 K 2023-03-23 19:54:43,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c84b22c5511242a1913d8a8c56211327 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327 2023-03-23 19:54:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:43,712 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:43,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:43,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:43,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,713 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:43,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:43,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:43,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:43,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:43,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:43,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327, entries=150, sequenceid=128, filesize=11.6 K 2023-03-23 19:54:43,733 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 18eab3ab7700f84be1d9a1a02e88ccfa in 885ms, sequenceid=128, compaction requested=true 2023-03-23 19:54:43,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:43,733 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:43,733 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:43,733 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:43,733 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:43,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:43,735 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:43,735 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:43,735 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,735 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6c818906d8ce4beba73b2818c1f1bc84, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=37.2 K 2023-03-23 19:54:43,736 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:43,736 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:43,736 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,736 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/42128d080f0d4825ac1f0781effb0246, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=34.9 K 2023-03-23 19:54:43,736 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6c818906d8ce4beba73b2818c1f1bc84, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:43,741 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 601245484906478990c1f26c78dddaed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679601281727 2023-03-23 19:54:43,741 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 42128d080f0d4825ac1f0781effb0246, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:43,745 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 960af9037afd440581f5cbe382f08d7b, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:43,745 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9d7266bc46e541c6a87d8e1823cded28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679601281727 2023-03-23 19:54:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:43,746 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c9d34a409c5d453288f4f46c480caa10, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:43,746 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,746 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:43,746 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-23 19:54:43,746 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:43,747 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:43,787 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#208 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:43,799 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:43,846 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/59d1a9cad0824a4982482cd8ccb45e72 2023-03-23 19:54:43,858 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/81dac31617b64d3281b98e7175353ee2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/81dac31617b64d3281b98e7175353ee2 2023-03-23 19:54:43,863 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 81dac31617b64d3281b98e7175353ee2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:43,863 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:43,863 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601283733; duration=0sec 2023-03-23 19:54:43,864 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:43,864 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:43,865 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:43,865 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:43,865 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:43,865 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/657a77d45f4b49be9ca350655a98fe8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=34.9 K 2023-03-23 19:54:43,866 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 657a77d45f4b49be9ca350655a98fe8b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1679601281186 2023-03-23 19:54:43,866 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7d817cfa025b403a93f064698c434ebd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679601281727 2023-03-23 19:54:43,866 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c84b22c5511242a1913d8a8c56211327, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:43,869 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/81548c5352f84267a053cec7965ed7db 2023-03-23 19:54:43,877 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:43,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:44,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601344005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601344006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601344109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601344109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,252 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/18d8e64fd4524e55b689607b00ea1187 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/18d8e64fd4524e55b689607b00ea1187 2023-03-23 19:54:44,256 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 18d8e64fd4524e55b689607b00ea1187(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:44,256 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:44,257 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601283733; duration=0sec 2023-03-23 19:54:44,257 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:44,292 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/3df710b570f04566a71889c0e2308513 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/3df710b570f04566a71889c0e2308513 2023-03-23 19:54:44,296 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a553bd830fe94b7c9b6f01b88e47b187 2023-03-23 19:54:44,302 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 3df710b570f04566a71889c0e2308513(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:44,302 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:44,302 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601283734; duration=0sec 2023-03-23 19:54:44,302 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:44,304 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/59d1a9cad0824a4982482cd8ccb45e72 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72 2023-03-23 19:54:44,308 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72, entries=150, sequenceid=143, filesize=11.8 K 2023-03-23 19:54:44,308 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/81548c5352f84267a053cec7965ed7db as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db 2023-03-23 19:54:44,312 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db, entries=150, sequenceid=143, filesize=11.8 K 2023-03-23 19:54:44,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,313 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a553bd830fe94b7c9b6f01b88e47b187 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187 2023-03-23 19:54:44,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601344311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601344311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,317 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187, entries=150, sequenceid=143, filesize=11.8 K 2023-03-23 19:54:44,318 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 18eab3ab7700f84be1d9a1a02e88ccfa in 572ms, sequenceid=143, compaction requested=false 2023-03-23 19:54:44,318 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:44,318 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:44,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@151dd4b4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:44,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:44,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,334 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:44,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:44,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:44,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:44,338 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:44,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:44,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:44,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:44,338 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:44,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,341 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:44,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:44,351 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:44,351 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:44,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:54:44,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:44,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:44,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:44,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:44,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:44,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:44,451 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:44,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:44,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:44,457 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:44,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:44,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:44,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:44,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:44,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:44,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:44,463 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,463 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:44,463 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:44,463 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:44,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:44,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d27a477[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:44,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:44,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:44,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:44,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601344465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601344468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,470 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601344469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:44,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:44,473 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:44,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:44,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:44,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:44,473 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:44,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,478 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,481 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:44,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:44,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,482 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:44,482 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:44,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601344570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601344571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601344572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,582 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:44,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:44,586 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:44,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:44,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:44,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:44,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:44,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:44,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:44,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,593 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,594 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,594 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:44,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5c17c8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:44,596 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:44,596 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:44,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:44,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:44,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,599 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:44,602 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:44,602 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:44,602 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:44,602 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:44,602 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:44,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,605 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,609 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:44,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:44,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,613 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,613 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:44,613 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601344614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601344616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,713 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:44,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:44,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:44,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:44,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:44,717 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:44,717 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,717 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:44,717 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:44,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,719 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:44,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:44,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:44,721 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,721 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:44,721 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:44,722 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:44,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,724 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5f77de[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:44,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:44,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,728 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,728 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:44,728 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:44,728 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:44,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:44,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:44,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:44,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:44,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:44,740 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:44,740 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:44,740 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:44,740 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:44,740 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:44,742 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,742 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,742 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:44,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:44,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,750 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:44,750 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:44,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601344774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601344774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601344774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,850 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:44,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:44,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:44,854 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:44,854 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:44,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:44,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/0162beca5dbf4b33aa67952cec3e6711 2023-03-23 19:54:44,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:44,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:44,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:44,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:44,864 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:44,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:44,867 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:44,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@351cb2d5[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:44,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:44,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:44,872 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,872 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:44,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:44,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,873 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:44,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:44,876 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:44,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:44,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:44,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:44,876 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:44,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:44,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:44,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,883 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:44,883 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:44,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:44,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:44,884 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:44,884 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:44,884 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:44,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:44,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/d59c31a0d9fd4776bdf0557a5e48e266 2023-03-23 19:54:44,984 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:44,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:44,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:44,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:44,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:44,998 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:44,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:44,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:44,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:44,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:44,999 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:45,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:45,002 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,002 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:45,002 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:45,002 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:45,004 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5578bc1d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:45,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:45,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,008 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:45,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,009 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:45,012 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:45,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:45,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:45,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:45,012 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:45,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:45,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:45,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,018 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:45,018 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601345078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601345079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601345079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,119 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:45,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:45,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601345118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601345119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:45,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:45,122 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:45,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:45,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:45,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:45,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:45,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:45,126 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,127 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:45,127 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:45,127 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:45,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:45,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:45,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:45,128 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:45,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd0132d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:45,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:45,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:45,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:45,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:45,132 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:45,133 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:45,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:45,136 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:45,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:45,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:45,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:45,136 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:45,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,141 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,143 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:45,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,143 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,143 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:45,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,143 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,244 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:45,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:45,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:45,247 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:45,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:45,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:45,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,251 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,252 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,252 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:45,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:45,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:45,253 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:45,253 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:45,254 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,254 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:45,254 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:45,254 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:45,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:45,256 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:45,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a84668e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:45,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:45,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,259 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:45,259 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:45,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:45,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:45,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:45,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:45,265 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:45,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:45,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:45,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:45,265 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:45,266 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,267 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,267 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,267 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,269 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,272 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,272 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:45,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:45,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,273 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:45,273 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9606b5457c0a4f5a8484215e02438d8d 2023-03-23 19:54:45,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/0162beca5dbf4b33aa67952cec3e6711 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711 2023-03-23 19:54:45,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711, entries=200, sequenceid=164, filesize=14.1 K 2023-03-23 19:54:45,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/d59c31a0d9fd4776bdf0557a5e48e266 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266 2023-03-23 19:54:45,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266, entries=150, sequenceid=164, filesize=11.8 K 2023-03-23 19:54:45,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9606b5457c0a4f5a8484215e02438d8d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d 2023-03-23 19:54:45,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d, entries=150, sequenceid=164, filesize=11.8 K 2023-03-23 19:54:45,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 18eab3ab7700f84be1d9a1a02e88ccfa in 885ms, sequenceid=164, compaction requested=true 2023-03-23 19:54:45,333 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:45,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:45,333 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:45,333 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:45,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:45,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:45,334 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:45,335 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:45,335 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:45,335 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,335 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/18d8e64fd4524e55b689607b00ea1187, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=35.3 K 2023-03-23 19:54:45,335 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:45,335 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,335 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/81dac31617b64d3281b98e7175353ee2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=37.7 K 2023-03-23 19:54:45,336 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 18d8e64fd4524e55b689607b00ea1187, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:45,336 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 81dac31617b64d3281b98e7175353ee2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:45,337 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 81548c5352f84267a053cec7965ed7db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679601282861 2023-03-23 19:54:45,337 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 59d1a9cad0824a4982482cd8ccb45e72, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679601282861 2023-03-23 19:54:45,337 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d59c31a0d9fd4776bdf0557a5e48e266, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:45,338 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 0162beca5dbf4b33aa67952cec3e6711, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:45,365 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:45,368 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#217 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:45,373 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:45,380 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:45,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:45,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:45,381 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,381 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:45,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:45,412 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,412 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:45,413 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:45,413 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:45,422 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/76c2c6a06df040d0b39b63ccbedcdc32 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/76c2c6a06df040d0b39b63ccbedcdc32 2023-03-23 19:54:45,428 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 76c2c6a06df040d0b39b63ccbedcdc32(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:45,428 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:45,428 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601285333; duration=0sec 2023-03-23 19:54:45,428 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:45,428 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:45,430 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:45,430 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:45,430 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,430 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/3df710b570f04566a71889c0e2308513, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=35.3 K 2023-03-23 19:54:45,430 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4f80c8897f6c43b38ae1e35d047dbdd2 2023-03-23 19:54:45,430 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3df710b570f04566a71889c0e2308513, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679601282322 2023-03-23 19:54:45,431 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a553bd830fe94b7c9b6f01b88e47b187, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679601282861 2023-03-23 19:54:45,431 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9606b5457c0a4f5a8484215e02438d8d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:45,448 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#220 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:45,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:45,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:45,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601345599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601345602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601345602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601345703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601345706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601345706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,797 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/e59dd7acd2bf4952b9d781b849f6dbd3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e59dd7acd2bf4952b9d781b849f6dbd3 2023-03-23 19:54:45,802 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into e59dd7acd2bf4952b9d781b849f6dbd3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:45,803 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:45,803 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601285333; duration=0sec 2023-03-23 19:54:45,803 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:45,854 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/3a3de4088c274a989e07cdcf9c6b9fb6 2023-03-23 19:54:45,868 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/80648660a012498ca14185876c4d5da9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/80648660a012498ca14185876c4d5da9 2023-03-23 19:54:45,874 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 80648660a012498ca14185876c4d5da9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:45,874 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:45,874 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601285333; duration=0sec 2023-03-23 19:54:45,874 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:45,876 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fdd49e9339b549f6bb139fd5dd4fc1dd 2023-03-23 19:54:45,883 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4f80c8897f6c43b38ae1e35d047dbdd2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2 2023-03-23 19:54:45,889 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2, entries=150, sequenceid=178, filesize=11.8 K 2023-03-23 19:54:45,891 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/3a3de4088c274a989e07cdcf9c6b9fb6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6 2023-03-23 19:54:45,906 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6, entries=150, sequenceid=178, filesize=11.8 K 2023-03-23 19:54:45,907 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fdd49e9339b549f6bb139fd5dd4fc1dd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd 2023-03-23 19:54:45,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601345906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601345910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:45,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601345911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,937 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd, entries=150, sequenceid=178, filesize=11.8 K 2023-03-23 19:54:45,938 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 18eab3ab7700f84be1d9a1a02e88ccfa in 525ms, sequenceid=178, compaction requested=false 2023-03-23 19:54:45,938 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:45,938 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:45,938 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:45,938 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:45,939 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:45,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,942 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:45,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a5ef311[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:45,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:45,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:45,946 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:45,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:45,948 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:45,951 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:45,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:45,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:45,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:45,952 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:45,954 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,954 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,954 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:45,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:45,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:45,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:45,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:45,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:45,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:45,965 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:45,965 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:45,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:45,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:45,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,066 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:46,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:46,072 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:46,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:46,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:46,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,074 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:46,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:46,092 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:46,092 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:46,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:46,104 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:54:46,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:46,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:46,105 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:46,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:46,134 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/656e8568ce9c4e8fafe47244771f47e0 2023-03-23 19:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601346151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601346153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601346212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601346213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601346213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601346255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601346259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601346459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601346463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,579 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/a6451500a43840ff98ebaf24c0428b0e 2023-03-23 19:54:46,592 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d95ce270c47343b19b4386891b95829d 2023-03-23 19:54:46,595 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/656e8568ce9c4e8fafe47244771f47e0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0 2023-03-23 19:54:46,606 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0, entries=150, sequenceid=200, filesize=11.8 K 2023-03-23 19:54:46,607 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/a6451500a43840ff98ebaf24c0428b0e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e 2023-03-23 19:54:46,610 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e, entries=150, sequenceid=200, filesize=11.8 K 2023-03-23 19:54:46,611 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d95ce270c47343b19b4386891b95829d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d 2023-03-23 19:54:46,615 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d, entries=150, sequenceid=200, filesize=11.8 K 2023-03-23 19:54:46,616 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 18eab3ab7700f84be1d9a1a02e88ccfa in 512ms, sequenceid=200, compaction requested=true 2023-03-23 19:54:46,616 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:46,616 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:46,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:46,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad63013[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:46,621 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:46,621 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:46,623 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,623 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:46,625 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:46,628 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:46,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:46,628 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:46,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,636 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:46,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:46,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:46,636 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,637 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:46,637 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:46,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:46,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/9ccb1d4634b0425786253a48f2e6e320 2023-03-23 19:54:46,737 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:46,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:46,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:46,742 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:46,742 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,742 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:46,742 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:46,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,744 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,744 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,744 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:46,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601346743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601346744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601346745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:46,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:46,752 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,752 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:46,752 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:46,753 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:46,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75800f1f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:46,757 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:46,757 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,762 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,762 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,762 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:46,765 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601346764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601346764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:46,767 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:46,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:46,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:46,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:46,768 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:46,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,770 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,775 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,775 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:46,775 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,775 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,776 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,776 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:46,776 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:46,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601346846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601346848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:46,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601346849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,876 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:46,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:46,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:46,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:46,879 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:46,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:46,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:46,880 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,880 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,881 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:46,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:46,883 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,883 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:46,883 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:46,883 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:46,885 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c39a737[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:46,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:46,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,888 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:46,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,888 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:46,890 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:46,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:46,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:46,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:46,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:46,895 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:46,898 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:46,898 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:46,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:46,899 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:46,899 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:46,999 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:46,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,003 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,005 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,005 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,005 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,005 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,008 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:47,011 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717215da[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:47,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:47,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:47,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:47,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:47,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:47,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:47,022 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:47,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:47,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:47,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:47,022 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:47,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,026 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:47,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,031 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:47,031 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:47,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:47,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601347049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601347051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601347052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,131 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:47,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,135 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,137 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,139 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,139 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,139 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,139 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:47,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:47,142 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/24a6cbdf969740d393403be0e85dcd8a 2023-03-23 19:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afff54f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:47,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:47,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:47,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:47,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:47,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:47,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:47,158 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:47,158 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:47,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,161 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,161 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,163 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:47,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,166 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,166 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:47,166 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,167 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,167 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,166 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:47,267 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:47,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601347269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,273 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,274 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601347272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,276 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,278 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,278 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,278 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,278 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:47,280 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,280 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,281 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:47,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f72fff8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:47,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:47,283 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:47,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,285 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:47,287 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:47,294 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:47,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:47,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:47,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:47,294 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:47,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,307 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:47,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,313 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:47,313 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:47,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601347353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601347355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601347355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,413 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:47,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,416 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,416 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,416 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,416 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,419 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,424 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:47,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:47,427 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec3eb3e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:47,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:47,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,432 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:47,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:47,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:47,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:47,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:47,437 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:47,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:47,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:47,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:47,437 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:47,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:47,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:47,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,446 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:47,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,446 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:47,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,546 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:47,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,549 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,554 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,554 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,554 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,554 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:47,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:47,556 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:47,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7eeff9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:47,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:47,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/35d8060e4e554c37bbd00fb63108ba46 2023-03-23 19:54:47,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:47,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:47,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:47,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/9ccb1d4634b0425786253a48f2e6e320 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320 2023-03-23 19:54:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320, entries=200, sequenceid=215, filesize=14.1 K 2023-03-23 19:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:47,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/24a6cbdf969740d393403be0e85dcd8a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a 2023-03-23 19:54:47,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:47,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:47,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:47,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:47,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:47,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:47,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:47,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a, entries=150, sequenceid=215, filesize=11.8 K 2023-03-23 19:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:47,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/35d8060e4e554c37bbd00fb63108ba46 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46 2023-03-23 19:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46, entries=150, sequenceid=215, filesize=11.8 K 2023-03-23 19:54:47,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 18eab3ab7700f84be1d9a1a02e88ccfa in 862ms, sequenceid=215, compaction requested=true 2023-03-23 19:54:47,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:47,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:47,583 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:47,583 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:47,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:47,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:47,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,583 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:47,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:47,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:47,583 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:47,584 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50842 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:47,584 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:47,584 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,584 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e59dd7acd2bf4952b9d781b849f6dbd3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=49.7 K 2023-03-23 19:54:47,585 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:47,585 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e59dd7acd2bf4952b9d781b849f6dbd3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:47,585 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:47,585 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,585 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/76c2c6a06df040d0b39b63ccbedcdc32, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=47.3 K 2023-03-23 19:54:47,586 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4f80c8897f6c43b38ae1e35d047dbdd2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679601284460 2023-03-23 19:54:47,587 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 656e8568ce9c4e8fafe47244771f47e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1679601285587 2023-03-23 19:54:47,588 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 76c2c6a06df040d0b39b63ccbedcdc32, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:47,589 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9ccb1d4634b0425786253a48f2e6e320, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286134 2023-03-23 19:54:47,589 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3a3de4088c274a989e07cdcf9c6b9fb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679601284460 2023-03-23 19:54:47,590 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a6451500a43840ff98ebaf24c0428b0e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1679601285587 2023-03-23 19:54:47,590 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 24a6cbdf969740d393403be0e85dcd8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286151 2023-03-23 19:54:47,610 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:47,613 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:47,639 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/e2c6a16b17b54a8abc16da1853ad3955 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e2c6a16b17b54a8abc16da1853ad3955 2023-03-23 19:54:47,643 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into e2c6a16b17b54a8abc16da1853ad3955(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:47,643 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:47,643 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=12, startTime=1679601287583; duration=0sec 2023-03-23 19:54:47,643 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:47,643 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:47,644 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:47,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:47,645 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,645 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/80648660a012498ca14185876c4d5da9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=47.3 K 2023-03-23 19:54:47,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 80648660a012498ca14185876c4d5da9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679601284001 2023-03-23 19:54:47,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fdd49e9339b549f6bb139fd5dd4fc1dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679601284460 2023-03-23 19:54:47,646 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d95ce270c47343b19b4386891b95829d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1679601285587 2023-03-23 19:54:47,646 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 35d8060e4e554c37bbd00fb63108ba46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286151 2023-03-23 19:54:47,653 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#230 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:47,661 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/ab7e882597074f679ecfcfba336421a7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ab7e882597074f679ecfcfba336421a7 2023-03-23 19:54:47,665 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into ab7e882597074f679ecfcfba336421a7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:47,665 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:47,665 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=12, startTime=1679601287583; duration=0sec 2023-03-23 19:54:47,665 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:47,684 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:47,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:47,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:47,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:47,687 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:47,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:47,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:47,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:47,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:47,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:47,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:47,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:47,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:47,691 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:47,691 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:47,701 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/66edee93c1aa4b2b883ac35f48bdf7ec 2023-03-23 19:54:47,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:47,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:47,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601347873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601347876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601347876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601347979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601347992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:47,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:47,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601347993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,039 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/792ba1ad96164fda9d2458bf8ae3191d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/792ba1ad96164fda9d2458bf8ae3191d 2023-03-23 19:54:48,043 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 792ba1ad96164fda9d2458bf8ae3191d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:48,044 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:48,044 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=12, startTime=1679601287583; duration=0sec 2023-03-23 19:54:48,044 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:48,115 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/8c0eeccd17764faaaf016a642e755065 2023-03-23 19:54:48,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601348181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601348196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601348197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601348278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601348284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601348484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601348500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601348500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,525 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fc5c55602a2146448899ac43be595717 2023-03-23 19:54:48,529 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/66edee93c1aa4b2b883ac35f48bdf7ec as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec 2023-03-23 19:54:48,533 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec, entries=150, sequenceid=234, filesize=11.8 K 2023-03-23 19:54:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/8c0eeccd17764faaaf016a642e755065 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065 2023-03-23 19:54:48,538 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065, entries=150, sequenceid=234, filesize=11.8 K 2023-03-23 19:54:48,539 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fc5c55602a2146448899ac43be595717 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717 2023-03-23 19:54:48,543 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717, entries=150, sequenceid=234, filesize=11.8 K 2023-03-23 19:54:48,544 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 853ms, sequenceid=234, compaction requested=false 2023-03-23 19:54:48,544 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:48,544 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:48,546 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5afe26d4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:48,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:48,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,549 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:48,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:48,551 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:48,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:48,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:48,554 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:48,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:48,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:48,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:48,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:48,562 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:48,562 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:48,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:48,563 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:48,662 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:48,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:48,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:48,665 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:48,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:48,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:48,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:48,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:48,667 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:48,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:48,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:48,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:48,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:48,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:48,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:48,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:48,669 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:48,669 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:48,669 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:48,669 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:48,669 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:48,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:48,677 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/65d506d2d1a041ddbc069b352582ae25 2023-03-23 19:54:48,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:49,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601349012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,088 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/c6f5ff9cb80c4d64a761e8618dc99cf9 2023-03-23 19:54:49,103 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c331aa783bad440c8346f01568465a23 2023-03-23 19:54:49,112 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/65d506d2d1a041ddbc069b352582ae25 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25 2023-03-23 19:54:49,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601349115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25, entries=150, sequenceid=251, filesize=11.8 K 2023-03-23 19:54:49,118 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/c6f5ff9cb80c4d64a761e8618dc99cf9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9 2023-03-23 19:54:49,123 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9, entries=150, sequenceid=251, filesize=11.8 K 2023-03-23 19:54:49,124 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c331aa783bad440c8346f01568465a23 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23 2023-03-23 19:54:49,128 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23, entries=150, sequenceid=251, filesize=11.8 K 2023-03-23 19:54:49,128 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 459ms, sequenceid=251, compaction requested=true 2023-03-23 19:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:49,129 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:49,130 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12bb9f4c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:49,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:49,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,134 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:49,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:49,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:49,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:49,139 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:49,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:49,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:49,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:49,140 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:49,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:49,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:49,142 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,143 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:49,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:49,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:49,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:49,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:49,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:49,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:49,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:49,146 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:49,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:49,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,147 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:49,147 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:49,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:49,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,247 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:49,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:49,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:49,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:49,250 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:49,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:49,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:49,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:49,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,251 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:49,253 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:49,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:49,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:49,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:49,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:49,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:49,255 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:49,255 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:49,256 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:49,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:49,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601349334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46510 deadline: 1679601349438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601349644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,663 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/6187ab99f2c24ed29fb283724257bca2 2023-03-23 19:54:49,680 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b25cccb7584747078d940be4391af6dc 2023-03-23 19:54:49,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601349947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601349948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:49,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601349949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,091 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d264b94158ce43efb1df7cbd75a0bf85 2023-03-23 19:54:50,095 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/6187ab99f2c24ed29fb283724257bca2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2 2023-03-23 19:54:50,099 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2, entries=150, sequenceid=267, filesize=11.9 K 2023-03-23 19:54:50,100 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b25cccb7584747078d940be4391af6dc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc 2023-03-23 19:54:50,103 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc, entries=150, sequenceid=267, filesize=11.9 K 2023-03-23 19:54:50,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d264b94158ce43efb1df7cbd75a0bf85 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85 2023-03-23 19:54:50,107 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85, entries=150, sequenceid=267, filesize=11.9 K 2023-03-23 19:54:50,108 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 852ms, sequenceid=267, compaction requested=true 2023-03-23 19:54:50,108 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:50,108 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:50,108 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:50,109 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:50,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d00c3e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:50,111 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:50,111 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,112 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:50,113 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:50,113 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,113 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:50,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,114 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:50,117 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:50,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:50,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:50,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:50,117 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:50,120 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:50,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:50,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:50,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:50,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:50,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:50,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:50,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:50,126 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:50,127 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:50,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:50,227 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:50,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:50,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:50,230 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:50,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:50,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:50,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:50,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,232 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:50,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:50,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:50,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:50,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:50,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:50,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:50,234 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:50,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,242 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/d81470317abe47c0afa06dec11bfa265 2023-03-23 19:54:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:50,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:50,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601350332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601350333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601350438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601350438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601350451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601350454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601350454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601350641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601350642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,652 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/86e0ae8682f64dd286d65d5e4b09d878 2023-03-23 19:54:50,662 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/7ece86f5263748138ca0bf305ffb7427 2023-03-23 19:54:50,666 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/d81470317abe47c0afa06dec11bfa265 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265 2023-03-23 19:54:50,669 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265, entries=150, sequenceid=283, filesize=11.9 K 2023-03-23 19:54:50,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/86e0ae8682f64dd286d65d5e4b09d878 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878 2023-03-23 19:54:50,674 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878, entries=150, sequenceid=283, filesize=11.9 K 2023-03-23 19:54:50,674 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/7ece86f5263748138ca0bf305ffb7427 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427 2023-03-23 19:54:50,678 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427, entries=150, sequenceid=283, filesize=11.9 K 2023-03-23 19:54:50,679 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 445ms, sequenceid=283, compaction requested=true 2023-03-23 19:54:50,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:50,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:50,685 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba9269a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:50,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:50,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:50,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:50,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:50,688 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,688 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:50,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:50,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,690 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:50,692 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:50,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:50,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:50,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:50,692 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:50,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:50,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:50,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:50,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:50,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:50,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:50,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:50,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,699 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:50,699 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:50,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:50,799 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:50,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:50,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:50,803 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:50,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:50,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:50,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:50,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:50,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:50,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:50,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:50,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:50,806 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:50,807 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:50,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:50,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601350975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601350978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601351081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601351082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,218 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/597d6fe48159471bbce18b61d006ccd6 2023-03-23 19:54:51,228 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/3e0ae19323db44009794fa9140047d7b 2023-03-23 19:54:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601351284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601351285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601351460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601351463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601351465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46504 deadline: 1679601351586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:51,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46528 deadline: 1679601351589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,638 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/ca076b452ee048c4b82ff081165632d0 2023-03-23 19:54:51,643 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/597d6fe48159471bbce18b61d006ccd6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6 2023-03-23 19:54:51,646 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6, entries=150, sequenceid=299, filesize=11.9 K 2023-03-23 19:54:51,647 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/3e0ae19323db44009794fa9140047d7b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b 2023-03-23 19:54:51,653 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b, entries=150, sequenceid=299, filesize=11.9 K 2023-03-23 19:54:51,659 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/ca076b452ee048c4b82ff081165632d0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0 2023-03-23 19:54:51,663 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0, entries=150, sequenceid=299, filesize=11.9 K 2023-03-23 19:54:51,664 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 18eab3ab7700f84be1d9a1a02e88ccfa in 857ms, sequenceid=299, compaction requested=true 2023-03-23 19:54:51,664 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:51,664 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:51,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ca6bbd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:51,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:51,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,669 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:51,670 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,670 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:51,672 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:51,676 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:51,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:51,676 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:51,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:51,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:51,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,683 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:51,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,684 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:51,784 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:51,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:51,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:51,787 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:51,788 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,788 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,789 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:51,789 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:51,789 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,789 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:51,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:51,791 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:51,791 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:51,799 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/87af869eac384364b650f3f78ce25f0b 2023-03-23 19:54:51,809 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/2a39e0c87571491d8da7266b560a0e51 2023-03-23 19:54:51,819 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5 2023-03-23 19:54:51,824 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/87af869eac384364b650f3f78ce25f0b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b 2023-03-23 19:54:51,828 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b, entries=150, sequenceid=315, filesize=11.9 K 2023-03-23 19:54:51,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/2a39e0c87571491d8da7266b560a0e51 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51 2023-03-23 19:54:51,837 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51, entries=150, sequenceid=315, filesize=11.9 K 2023-03-23 19:54:51,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5 2023-03-23 19:54:51,841 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5, entries=150, sequenceid=315, filesize=11.9 K 2023-03-23 19:54:51,842 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 18eab3ab7700f84be1d9a1a02e88ccfa in 51ms, sequenceid=315, compaction requested=true 2023-03-23 19:54:51,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:51,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:51,844 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec5c317[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:51,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:51,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,848 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:51,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:51,850 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:51,852 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:51,852 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:51,853 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:51,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:51,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:51,858 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:51,858 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,859 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:51,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:51,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,959 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:51,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:51,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:51,962 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:51,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:51,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:51,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:51,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:51,968 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:51,968 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,968 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:51,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:51,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:51,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:51,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:51,971 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,976 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d97910d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:51,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:51,979 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:51,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:51,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:51,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:51,984 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:51,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:51,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:51,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:51,984 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:51,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:51,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:51,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:51,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:51,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:51,992 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:51,992 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:51,992 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:52,092 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:52,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:52,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:52,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:52,096 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:52,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:52,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:52,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:52,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,096 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:52,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,114 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:52,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:52,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:52,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:52,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:52,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:52,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:52,116 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:52,116 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:52,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:52,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:52,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601292146, max timestamp=1679601292160Num uniques 859; 2023-03-23 19:54:52,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:52,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601292146, max timestamp=1679601292160Num uniques 859; 2023-03-23 19:54:52,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:52,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601292146, max timestamp=1679601292160Num uniques 859; 2023-03-23 19:54:52,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:52,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601292196, max timestamp=1679601292208Num uniques 1018; 2023-03-23 19:54:52,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:52,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601292196, max timestamp=1679601292208Num uniques 1018; 2023-03-23 19:54:52,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679601292196, max timestamp=1679601292208Num uniques 1018; 2023-03-23 19:54:52,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:52,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-23 19:54:52,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601292232, max timestamp=1679601292253Num uniques 1077; 2023-03-23 19:54:52,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-23 19:54:52,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601292232, max timestamp=1679601292253Num uniques 1077; 2023-03-23 19:54:52,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-23 19:54:52,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679601292232, max timestamp=1679601292253Num uniques 1077; 2023-03-23 19:54:52,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679601292277, max timestamp=1679601292295Num uniques 1036; 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679601292277, max timestamp=1679601292295Num uniques 1036; 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679601292277, max timestamp=1679601292295Num uniques 1036; 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-23 19:54:52,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601292340, max timestamp=1679601292350Num uniques 1095; 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601292340, max timestamp=1679601292350Num uniques 1095; 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679601292340, max timestamp=1679601292350Num uniques 1095; 2023-03-23 19:54:52,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679601292374, max timestamp=1679601292389Num uniques 1104; 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679601292374, max timestamp=1679601292389Num uniques 1104; 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679601292374, max timestamp=1679601292389Num uniques 1104; 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-23 19:54:52,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679601292418, max timestamp=1679601292429Num uniques 1063; 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679601292418, max timestamp=1679601292429Num uniques 1063; 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679601292418, max timestamp=1679601292429Num uniques 1063; 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-23 19:54:52,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679601292470, max timestamp=1679601292484Num uniques 1072; 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679601292470, max timestamp=1679601292484Num uniques 1072; 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679601292470, max timestamp=1679601292484Num uniques 1072; 2023-03-23 19:54:52,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-23 19:54:52,526 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/3bbd9bd7fde3491ea3f9e12bf2994ffb 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601292523, max timestamp=1679601292537Num uniques 1031; 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601292523, max timestamp=1679601292537Num uniques 1031; 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679601292523, max timestamp=1679601292537Num uniques 1031; 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-23 19:54:52,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-23 19:54:52,543 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/1ddc6acd65b64f368bfd8f802a698afb 2023-03-23 19:54:52,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679601292571, max timestamp=1679601292585Num uniques 1090; 2023-03-23 19:54:52,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-23 19:54:52,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679601292571, max timestamp=1679601292585Num uniques 1090; 2023-03-23 19:54:52,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-23 19:54:52,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679601292571, max timestamp=1679601292585Num uniques 1090; 2023-03-23 19:54:52,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679601292613, max timestamp=1679601292629Num uniques 1049; 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679601292613, max timestamp=1679601292629Num uniques 1049; 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679601292613, max timestamp=1679601292629Num uniques 1049; 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-23 19:54:52,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-23 19:54:52,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679601292662, max timestamp=1679601292674Num uniques 1058; 2023-03-23 19:54:52,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-23 19:54:52,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679601292662, max timestamp=1679601292674Num uniques 1058; 2023-03-23 19:54:52,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-23 19:54:52,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679601292662, max timestamp=1679601292674Num uniques 1058; 2023-03-23 19:54:52,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-23 19:54:52,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679601292694, max timestamp=1679601292707Num uniques 1067; 2023-03-23 19:54:52,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-23 19:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679601292694, max timestamp=1679601292707Num uniques 1067; 2023-03-23 19:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679601292694, max timestamp=1679601292707Num uniques 1067; 2023-03-23 19:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-23 19:54:52,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-23 19:54:52,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679601292732, max timestamp=1679601292749Num uniques 1076; 2023-03-23 19:54:52,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-23 19:54:52,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679601292732, max timestamp=1679601292749Num uniques 1076; 2023-03-23 19:54:52,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679601292732, max timestamp=1679601292749Num uniques 1076; 2023-03-23 19:54:52,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-23 19:54:52,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-23 19:54:52,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679601292782, max timestamp=1679601292791Num uniques 1035; 2023-03-23 19:54:52,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-23 19:54:52,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679601292782, max timestamp=1679601292791Num uniques 1035; 2023-03-23 19:54:52,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-23 19:54:52,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679601292782, max timestamp=1679601292791Num uniques 1035; 2023-03-23 19:54:52,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-23 19:54:52,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679601292819, max timestamp=1679601292833Num uniques 1044; 2023-03-23 19:54:52,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-23 19:54:52,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679601292819, max timestamp=1679601292833Num uniques 1044; 2023-03-23 19:54:52,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-23 19:54:52,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679601292819, max timestamp=1679601292833Num uniques 1044; 2023-03-23 19:54:52,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679601292863, max timestamp=1679601292884Num uniques 1053; 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679601292863, max timestamp=1679601292884Num uniques 1053; 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679601292863, max timestamp=1679601292884Num uniques 1053; 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-23 19:54:52,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-23 19:54:52,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679601292914, max timestamp=1679601292925Num uniques 1012; 2023-03-23 19:54:52,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-23 19:54:52,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679601292914, max timestamp=1679601292925Num uniques 1012; 2023-03-23 19:54:52,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-23 19:54:52,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679601292914, max timestamp=1679601292925Num uniques 1012; 2023-03-23 19:54:52,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-23 19:54:52,958 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/1f3467321cb0472c9f2d89b3cb62dee3 2023-03-23 19:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679601292947, max timestamp=1679601292960Num uniques 1071; 2023-03-23 19:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-23 19:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679601292947, max timestamp=1679601292960Num uniques 1071; 2023-03-23 19:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-23 19:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679601292947, max timestamp=1679601292960Num uniques 1071; 2023-03-23 19:54:52,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-23 19:54:52,966 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/3bbd9bd7fde3491ea3f9e12bf2994ffb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb 2023-03-23 19:54:52,970 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb, entries=150, sequenceid=323, filesize=11.9 K 2023-03-23 19:54:52,970 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/1ddc6acd65b64f368bfd8f802a698afb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb 2023-03-23 19:54:52,975 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb, entries=100, sequenceid=323, filesize=9.5 K 2023-03-23 19:54:52,976 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/1f3467321cb0472c9f2d89b3cb62dee3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3 2023-03-23 19:54:52,980 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3, entries=100, sequenceid=323, filesize=9.5 K 2023-03-23 19:54:52,981 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=67.09 KB/68700 for 18eab3ab7700f84be1d9a1a02e88ccfa in 865ms, sequenceid=323, compaction requested=true 2023-03-23 19:54:52,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:52,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:52,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=73.80 KB heapSize=199.88 KB 2023-03-23 19:54:52,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:52,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:52,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:52,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:52,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:52,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:52,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f58f9d3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:52,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:52,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,989 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:52,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:52,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:52,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:52,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:52,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:52,993 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:52,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:52,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:52,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:52,993 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:52,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:52,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:52,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:52,996 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:52,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:52,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:52,999 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:52,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:52,999 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:52,999 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:52,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:53,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601353034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601353036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,099 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,103 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,108 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,111 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2ce406[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,117 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,119 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,119 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,126 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,126 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601353139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601353141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,226 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,229 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,230 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,232 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,232 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,234 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c2157ef[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,238 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,239 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,242 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,242 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:53,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,248 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,248 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601353343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601353344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,348 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,352 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,355 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d9fbe9f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,365 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,365 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,366 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,368 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,371 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,371 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,377 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,381 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:53,382 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,382 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/cec50c940f6744b9811c95ac6b4193cd 2023-03-23 19:54:53,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601353471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,475 DEBUG [Thread-1477] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:53,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46506 deadline: 1679601353478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,481 DEBUG [Thread-1473] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:53,482 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,486 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601353484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,487 DEBUG [Thread-1479] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:54:53,487 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,487 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,488 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,491 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,492 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,492 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5581529e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,501 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,501 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,503 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,503 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,503 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,503 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,505 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,505 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,508 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,508 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,508 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,508 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,508 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,514 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,514 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,614 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,618 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,618 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,618 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,618 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,621 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,621 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,622 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,623 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,623 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,623 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,623 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,625 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4680fc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,628 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,629 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,635 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,637 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,638 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,638 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,638 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,638 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,641 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,644 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,644 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601353647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601353649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,744 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,747 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,750 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:53,751 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,751 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,753 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ae22f4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,756 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,756 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,757 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,757 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,757 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,759 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,759 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,759 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,761 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,761 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,764 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,764 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,764 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,765 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,766 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,769 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,770 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:53,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,770 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:53,770 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:53,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/bc14e526c859417581a170d4933a8b1c 2023-03-23 19:54:53,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c0d31060fa5a41ce99ea9758f2428192 2023-03-23 19:54:53,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/cec50c940f6744b9811c95ac6b4193cd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd 2023-03-23 19:54:53,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd, entries=250, sequenceid=660, filesize=16.7 K 2023-03-23 19:54:53,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/bc14e526c859417581a170d4933a8b1c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c 2023-03-23 19:54:53,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c, entries=150, sequenceid=660, filesize=11.9 K 2023-03-23 19:54:53,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/c0d31060fa5a41ce99ea9758f2428192 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192 2023-03-23 19:54:53,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192, entries=150, sequenceid=660, filesize=11.9 K 2023-03-23 19:54:53,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~219.75 KB/225024, currentSize=100.63 KB/103050 for 18eab3ab7700f84be1d9a1a02e88ccfa in 865ms, sequenceid=660, compaction requested=true 2023-03-23 19:54:53,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:53,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:53,846 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:53,846 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:53,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:53,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:53,852 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 114544 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:53,852 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:53,852 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,852 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e2c6a16b17b54a8abc16da1853ad3955, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=111.9 K 2023-03-23 19:54:53,852 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107224 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:53,852 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:53,852 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,853 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/792ba1ad96164fda9d2458bf8ae3191d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=104.7 K 2023-03-23 19:54:53,853 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e2c6a16b17b54a8abc16da1853ad3955, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286151 2023-03-23 19:54:53,853 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 792ba1ad96164fda9d2458bf8ae3191d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286151 2023-03-23 19:54:53,854 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 66edee93c1aa4b2b883ac35f48bdf7ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679601286739 2023-03-23 19:54:53,854 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8c0eeccd17764faaaf016a642e755065, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679601286739 2023-03-23 19:54:53,854 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 65d506d2d1a041ddbc069b352582ae25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601287867 2023-03-23 19:54:53,858 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6187ab99f2c24ed29fb283724257bca2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679601289005 2023-03-23 19:54:53,858 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c6f5ff9cb80c4d64a761e8618dc99cf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601287867 2023-03-23 19:54:53,858 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b25cccb7584747078d940be4391af6dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679601289005 2023-03-23 19:54:53,858 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d81470317abe47c0afa06dec11bfa265, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679601289326 2023-03-23 19:54:53,858 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 86e0ae8682f64dd286d65d5e4b09d878, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679601289326 2023-03-23 19:54:53,859 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 597d6fe48159471bbce18b61d006ccd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679601290311 2023-03-23 19:54:53,859 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3e0ae19323db44009794fa9140047d7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679601290311 2023-03-23 19:54:53,859 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 87af869eac384364b650f3f78ce25f0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679601290970 2023-03-23 19:54:53,859 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2a39e0c87571491d8da7266b560a0e51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679601290970 2023-03-23 19:54:53,860 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 3bbd9bd7fde3491ea3f9e12bf2994ffb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679601292096 2023-03-23 19:54:53,860 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 1ddc6acd65b64f368bfd8f802a698afb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679601292096 2023-03-23 19:54:53,860 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting cec50c940f6744b9811c95ac6b4193cd, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292968 2023-03-23 19:54:53,860 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bc14e526c859417581a170d4933a8b1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292975 2023-03-23 19:54:53,870 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:53,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:53,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:53,874 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:53,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:53,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:53,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,877 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:53,880 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#256 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:53,881 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#255 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:53,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:53,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:53,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:53,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:53,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:53,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:53,887 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,887 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:53,887 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:53,888 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:53,904 WARN [DataStreamer for file /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/8ccf379d30804874a5c1df3b39b034b8] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-23 19:54:53,912 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/8ccf379d30804874a5c1df3b39b034b8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8ccf379d30804874a5c1df3b39b034b8 2023-03-23 19:54:53,915 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/035d0b10ac714f87b8c9aceda9272307 2023-03-23 19:54:53,918 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 8ccf379d30804874a5c1df3b39b034b8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:53,918 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:53,918 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=7, startTime=1679601293846; duration=0sec 2023-03-23 19:54:53,918 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:53,919 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-23 19:54:53,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107224 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-23 19:54:53,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:53,932 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,932 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ab7e882597074f679ecfcfba336421a7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=104.7 K 2023-03-23 19:54:53,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ab7e882597074f679ecfcfba336421a7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1679601286151 2023-03-23 19:54:53,933 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fc5c55602a2146448899ac43be595717, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679601286739 2023-03-23 19:54:53,933 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c331aa783bad440c8346f01568465a23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679601287867 2023-03-23 19:54:53,934 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d264b94158ce43efb1df7cbd75a0bf85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679601289005 2023-03-23 19:54:53,934 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7ece86f5263748138ca0bf305ffb7427, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679601289326 2023-03-23 19:54:53,935 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ca076b452ee048c4b82ff081165632d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679601290311 2023-03-23 19:54:53,935 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 8bbe9447a7b24d53ac2cfbaae8a2c0b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679601290970 2023-03-23 19:54:53,935 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 1f3467321cb0472c9f2d89b3cb62dee3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679601292096 2023-03-23 19:54:53,936 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c0d31060fa5a41ce99ea9758f2428192, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292975 2023-03-23 19:54:53,941 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/5d95122a741e46598704e8d58e6be1f1 2023-03-23 19:54:53,954 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#260 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:53,964 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/447f9fd8c14e4166b234fb185fd1dd4d 2023-03-23 19:54:53,968 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/035d0b10ac714f87b8c9aceda9272307 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307 2023-03-23 19:54:53,974 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307, entries=150, sequenceid=678, filesize=11.9 K 2023-03-23 19:54:53,975 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/5d95122a741e46598704e8d58e6be1f1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1 2023-03-23 19:54:53,978 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1, entries=150, sequenceid=678, filesize=11.9 K 2023-03-23 19:54:53,979 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/447f9fd8c14e4166b234fb185fd1dd4d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d 2023-03-23 19:54:53,982 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d, entries=150, sequenceid=678, filesize=11.9 K 2023-03-23 19:54:53,983 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 18eab3ab7700f84be1d9a1a02e88ccfa in 96ms, sequenceid=678, compaction requested=false 2023-03-23 19:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:53,983 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,985 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e51238[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:53,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:53,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:53,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:53,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:53,993 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:53,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:53,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:53,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:53,993 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:53,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:53,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:53,996 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:54,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:54,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,001 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:54,001 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:54,101 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:54,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:54,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:54,106 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:54,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:54,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:54,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,108 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:54,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:54,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:54,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:54,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:54,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,113 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17afbc18[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:54,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:54,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,117 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:54,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:54,118 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:54,121 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:54,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:54,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:54,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:54,121 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:54,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,124 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:54,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,131 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:54,131 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:54,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:54,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:54,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4006dd92e6084fa1b97eb269a70d320e 2023-03-23 19:54:54,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601294187, max timestamp=1679601294211Num uniques 859; 2023-03-23 19:54:54,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601294187, max timestamp=1679601294211Num uniques 859; 2023-03-23 19:54:54,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:54,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:54,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601294187, max timestamp=1679601294211Num uniques 859; 2023-03-23 19:54:54,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:54,231 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:54,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:54,234 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:54,234 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,234 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:54,234 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:54,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,236 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:54,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:54,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:54,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:54,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:54,240 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:54,241 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d448f71[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:54,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:54,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:54,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:54,247 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:54,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:54,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:54,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:54,247 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:54,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,249 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,253 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:54,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,254 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:54,254 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:54,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601354249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601354254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,293 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b64c74f2bdb44a398bf8dfefc978e19d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b64c74f2bdb44a398bf8dfefc978e19d 2023-03-23 19:54:54,297 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into b64c74f2bdb44a398bf8dfefc978e19d(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:54,297 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:54,297 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=7, startTime=1679601293846; duration=0sec 2023-03-23 19:54:54,297 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:54,354 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:54,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:54,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:54,357 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:54,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:54,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:54,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601354355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,359 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:54,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:54,361 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:54,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601354360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,363 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2464b3af[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:54,365 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:54,365 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,366 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:54,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:54,368 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:54,371 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:54,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:54,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:54,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:54,371 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:54,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,374 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,375 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/5b7ce4811e4046f0a58b09a38df7e577 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5b7ce4811e4046f0a58b09a38df7e577 2023-03-23 19:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,376 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,381 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 5b7ce4811e4046f0a58b09a38df7e577(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:54,381 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:54,381 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=7, startTime=1679601293846; duration=0sec 2023-03-23 19:54:54,381 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:54,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:54,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:54,390 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:54,390 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:54,490 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:54,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:54,493 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:54,493 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,493 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:54,493 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:54,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,495 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,496 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,496 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:54,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:54,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:54,498 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,498 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:54,498 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:54,498 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:54,499 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f001c9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:54,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:54,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,506 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:54,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:54,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:54,511 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:54,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:54,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:54,511 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:54,511 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:54,512 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,512 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,513 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:54,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:54,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:54,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:54,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:54,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:54,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:54,520 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:54,520 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601354559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601354564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b09cdf5dfed543468d58704a9e1583e5 2023-03-23 19:54:54,607 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a6bb7457c6ad411db3dac342a850523d 2023-03-23 19:54:54,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4006dd92e6084fa1b97eb269a70d320e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e 2023-03-23 19:54:54,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e, entries=150, sequenceid=690, filesize=11.9 K 2023-03-23 19:54:54,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b09cdf5dfed543468d58704a9e1583e5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5 2023-03-23 19:54:54,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5, entries=150, sequenceid=690, filesize=11.9 K 2023-03-23 19:54:54,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/a6bb7457c6ad411db3dac342a850523d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d 2023-03-23 19:54:54,621 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:54,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d, entries=150, sequenceid=690, filesize=11.9 K 2023-03-23 19:54:54,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 18eab3ab7700f84be1d9a1a02e88ccfa in 455ms, sequenceid=690, compaction requested=true 2023-03-23 19:54:54,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:54,623 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:54,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-23 19:54:54,623 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:54,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:54,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:54,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:54,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:54,624 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:54,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:54,625 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,625 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:54,625 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8ccf379d30804874a5c1df3b39b034b8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.4 K 2023-03-23 19:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,625 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:54,625 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b64c74f2bdb44a398bf8dfefc978e19d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.4 K 2023-03-23 19:54:54,625 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8ccf379d30804874a5c1df3b39b034b8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292975 2023-03-23 19:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,625 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b64c74f2bdb44a398bf8dfefc978e19d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292975 2023-03-23 19:54:54,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 035d0b10ac714f87b8c9aceda9272307, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1679601293026 2023-03-23 19:54:54,626 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5d95122a741e46598704e8d58e6be1f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1679601293026 2023-03-23 19:54:54,626 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4006dd92e6084fa1b97eb269a70d320e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:54,626 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b09cdf5dfed543468d58704a9e1583e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:54,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,628 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:54,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,628 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:54,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:54,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:54,630 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:54,630 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:54,630 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:54,630 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-23 19:54:54,630 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:54,630 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:54,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:54,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:54,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:54,631 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:54,633 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#264 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:54,633 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:54,646 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/53b4040df2db4ff0be00004faff45e6d 2023-03-23 19:54:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:54,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:54,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601354874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601354875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601354977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:54,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:54,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601354979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,043 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/9d5876672a5e46759ccc4eb1001cb95c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d5876672a5e46759ccc4eb1001cb95c 2023-03-23 19:54:55,048 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 9d5876672a5e46759ccc4eb1001cb95c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:55,048 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:55,048 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601294623; duration=0sec 2023-03-23 19:54:55,048 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:55,048 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:55,049 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:55,049 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:55,049 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,050 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5b7ce4811e4046f0a58b09a38df7e577, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.4 K 2023-03-23 19:54:55,050 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5b7ce4811e4046f0a58b09a38df7e577, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1679601292975 2023-03-23 19:54:55,050 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/67bd2a5033714da4804799352f7b7e73 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/67bd2a5033714da4804799352f7b7e73 2023-03-23 19:54:55,051 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 447f9fd8c14e4166b234fb185fd1dd4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1679601293026 2023-03-23 19:54:55,054 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a6bb7457c6ad411db3dac342a850523d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:55,057 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 67bd2a5033714da4804799352f7b7e73(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:55,058 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:55,058 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601294623; duration=0sec 2023-03-23 19:54:55,058 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:55,058 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/8558de91c5fb4398876fee64a3ce8029 2023-03-23 19:54:55,064 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#268 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:55,080 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/5620f25acd954c63a7267c74d5d8f2d2 2023-03-23 19:54:55,085 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/65a4c046f56f449a91826db601a77469 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/65a4c046f56f449a91826db601a77469 2023-03-23 19:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/53b4040df2db4ff0be00004faff45e6d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d 2023-03-23 19:54:55,092 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d, entries=150, sequenceid=729, filesize=11.9 K 2023-03-23 19:54:55,093 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 65a4c046f56f449a91826db601a77469(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:55,094 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:55,094 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601294623; duration=0sec 2023-03-23 19:54:55,094 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:55,094 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/8558de91c5fb4398876fee64a3ce8029 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029 2023-03-23 19:54:55,098 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029, entries=150, sequenceid=729, filesize=11.9 K 2023-03-23 19:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/5620f25acd954c63a7267c74d5d8f2d2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2 2023-03-23 19:54:55,104 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2, entries=150, sequenceid=729, filesize=11.9 K 2023-03-23 19:54:55,104 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 18eab3ab7700f84be1d9a1a02e88ccfa in 474ms, sequenceid=729, compaction requested=false 2023-03-23 19:54:55,104 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:55,105 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,107 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,107 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,107 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e553a33[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,110 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,110 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,112 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,112 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,115 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,115 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,126 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,126 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:55,183 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:54:55,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:55,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:55,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:55,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:55,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:55,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:55,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/aa205c53abba4d8d816277551562733a 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601295205, max timestamp=1679601295216Num uniques 859; 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601295205, max timestamp=1679601295216Num uniques 859; 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601295205, max timestamp=1679601295216Num uniques 859; 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:55,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:55,226 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,235 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,235 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,236 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,237 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:55,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601355256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601355257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,268 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,270 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,271 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,271 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3773178f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,279 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,281 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,284 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,284 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,285 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,286 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,290 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,291 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,291 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:55,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601355361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601355361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,391 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,394 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,394 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:55,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,398 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,398 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,398 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,398 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,400 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,400 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360b3305[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,402 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,402 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,404 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,406 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,406 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,408 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,410 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,412 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,413 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,413 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:55,513 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,516 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,522 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,525 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72318244[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,527 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,527 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,528 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,528 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,528 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,529 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,532 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,532 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,534 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,534 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,534 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,535 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:55,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,541 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,541 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,541 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601355572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601355572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,641 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,644 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,644 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,644 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,644 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,645 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,645 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,646 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,646 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,646 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,646 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,647 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,647 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,647 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,648 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,648 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,649 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b2ba93[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,652 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,652 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,652 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,658 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,658 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,658 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,658 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,658 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,660 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,661 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/4ed9b0d451ce47098c28418d6742854a 2023-03-23 19:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,664 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,664 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,665 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,665 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,665 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:55,765 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,771 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,772 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,773 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,773 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,773 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:55,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,782 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a5204c5[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,784 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,784 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,785 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,786 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,786 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,787 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,790 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,790 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,792 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,792 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,792 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,800 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,800 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,800 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601355874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:55,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601355877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,900 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:55,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:55,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:55,903 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:55,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:55,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:55,904 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,904 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,905 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,905 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,906 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,906 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:55,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:55,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:55,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,912 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ae9791[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,919 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:55,920 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,920 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:55,921 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:55,924 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:55,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:55,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:55,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:55,924 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:55,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,927 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:55,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:55,931 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:55,931 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:55,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:56,031 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:56,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:56,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:56,034 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,037 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:56,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:56,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:56,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:56,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:56,038 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:56,039 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,039 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:56,039 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:56,039 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:56,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:56,041 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:56,041 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:56,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:56,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bf300c2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:56,042 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:56,042 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,043 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:56,043 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,043 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:56,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:56,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:56,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:56,047 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:56,047 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:56,047 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:56,047 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:56,047 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:56,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,049 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,049 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:56,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,053 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:56,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:56,053 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:56,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/eb474f64f2964e82834d5b063e8bf7ce 2023-03-23 19:54:56,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/aa205c53abba4d8d816277551562733a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a 2023-03-23 19:54:56,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a, entries=150, sequenceid=743, filesize=11.9 K 2023-03-23 19:54:56,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/4ed9b0d451ce47098c28418d6742854a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a 2023-03-23 19:54:56,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a, entries=150, sequenceid=743, filesize=11.9 K 2023-03-23 19:54:56,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/eb474f64f2964e82834d5b063e8bf7ce as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce 2023-03-23 19:54:56,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce, entries=150, sequenceid=743, filesize=11.9 K 2023-03-23 19:54:56,098 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 18eab3ab7700f84be1d9a1a02e88ccfa in 914ms, sequenceid=743, compaction requested=true 2023-03-23 19:54:56,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:56,098 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:56,098 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:56,098 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:56,098 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:56,098 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:56,099 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:56,099 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:56,099 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:56,099 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,099 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:56,099 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/67bd2a5033714da4804799352f7b7e73, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.5 K 2023-03-23 19:54:56,099 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,099 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d5876672a5e46759ccc4eb1001cb95c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.5 K 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 67bd2a5033714da4804799352f7b7e73, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9d5876672a5e46759ccc4eb1001cb95c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 53b4040df2db4ff0be00004faff45e6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1679601294226 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8558de91c5fb4398876fee64a3ce8029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1679601294226 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting aa205c53abba4d8d816277551562733a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:56,100 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4ed9b0d451ce47098c28418d6742854a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:56,106 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:56,106 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:56,133 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/f299a59f7cdc445f845c42e5e66f7b47 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/f299a59f7cdc445f845c42e5e66f7b47 2023-03-23 19:54:56,138 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into f299a59f7cdc445f845c42e5e66f7b47(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:56,138 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:56,138 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601296098; duration=0sec 2023-03-23 19:54:56,139 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:56,139 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:54:56,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:54:56,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:56,140 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,140 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/65a4c046f56f449a91826db601a77469, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.5 K 2023-03-23 19:54:56,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 65a4c046f56f449a91826db601a77469, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679601294161 2023-03-23 19:54:56,141 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5620f25acd954c63a7267c74d5d8f2d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=729, earliestPutTs=1679601294226 2023-03-23 19:54:56,141 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting eb474f64f2964e82834d5b063e8bf7ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:56,148 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:56,153 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:56,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:56,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:56,159 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:56,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:56,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:56,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:56,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:56,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:56,160 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:56,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:56,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:56,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:56,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:56,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:56,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:56,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:56,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:56,164 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:56,164 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:56,164 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:56,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:56,165 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,165 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:56,169 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/bcdcf524d77546e48992b9df3e7034fc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bcdcf524d77546e48992b9df3e7034fc 2023-03-23 19:54:56,174 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into bcdcf524d77546e48992b9df3e7034fc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:56,174 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:56,174 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601296098; duration=0sec 2023-03-23 19:54:56,174 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:56,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:56,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601356390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601356391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601356493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601356494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,517 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/34ba2c79cb9f46b896240a7bf8b3ab8b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/34ba2c79cb9f46b896240a7bf8b3ab8b 2023-03-23 19:54:56,522 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 34ba2c79cb9f46b896240a7bf8b3ab8b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:56,522 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:56,522 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601296098; duration=0sec 2023-03-23 19:54:56,522 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:56,577 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/3473108dc6cb418782e1b266893e8793 2023-03-23 19:54:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601356696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601356697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:56,987 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/686aa93b995740a9b166590da05d7ff2 2023-03-23 19:54:56,998 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/84f2ecba4cbf496f8a6a023925cd236f 2023-03-23 19:54:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601357000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,002 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/3473108dc6cb418782e1b266893e8793 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793 2023-03-23 19:54:57,006 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793, entries=150, sequenceid=780, filesize=11.9 K 2023-03-23 19:54:57,007 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/686aa93b995740a9b166590da05d7ff2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2 2023-03-23 19:54:57,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601357006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,012 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2, entries=150, sequenceid=780, filesize=11.9 K 2023-03-23 19:54:57,016 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/84f2ecba4cbf496f8a6a023925cd236f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f 2023-03-23 19:54:57,021 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f, entries=150, sequenceid=780, filesize=11.9 K 2023-03-23 19:54:57,022 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 18eab3ab7700f84be1d9a1a02e88ccfa in 858ms, sequenceid=780, compaction requested=false 2023-03-23 19:54:57,022 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:57,022 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:57,024 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf55d0d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:57,027 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:57,027 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:57,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:57,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:57,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:57,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:57,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:57,034 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:57,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:57,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:57,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:57,034 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:57,037 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,037 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:57,037 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:57,038 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:57,039 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:57,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:57,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:57,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,043 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,043 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:57,043 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,043 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:57,042 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:57,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:57,043 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:57,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:57,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:57,143 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:57,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:57,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:57,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:57,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:57,146 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:57,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:57,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:57,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:57,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:57,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,148 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:57,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:57,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,148 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:57,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:57,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:57,150 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:57,150 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:57,150 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:57,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:57,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:57,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601357532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679601297529, max timestamp=1679601297535Num uniques 859; 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679601297529, max timestamp=1679601297535Num uniques 859; 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679601297529, max timestamp=1679601297535Num uniques 859; 2023-03-23 19:54:57,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:54:57,559 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/5fa62f0474e9444f935a94ba8d944ac3 2023-03-23 19:54:57,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601357556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601357560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601357561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601357562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601357639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601357662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601357664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601357666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601357666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601357841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601357863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601357866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601357869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:57,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601357870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:57,971 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/6dfabef5fba24f40ab5e7e3c3a517ecb 2023-03-23 19:54:57,980 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fd9a29194cd94928b79478b1d9f9c548 2023-03-23 19:54:57,985 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/5fa62f0474e9444f935a94ba8d944ac3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3 2023-03-23 19:54:57,988 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3, entries=150, sequenceid=793, filesize=11.9 K 2023-03-23 19:54:57,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/6dfabef5fba24f40ab5e7e3c3a517ecb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb 2023-03-23 19:54:58,008 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb, entries=150, sequenceid=793, filesize=11.9 K 2023-03-23 19:54:58,010 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/fd9a29194cd94928b79478b1d9f9c548 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548 2023-03-23 19:54:58,028 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548, entries=150, sequenceid=793, filesize=11.9 K 2023-03-23 19:54:58,029 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.55 KB/134706 for 18eab3ab7700f84be1d9a1a02e88ccfa in 879ms, sequenceid=793, compaction requested=true 2023-03-23 19:54:58,029 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:58,029 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,031 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cb109a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,033 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,033 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,035 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,035 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,043 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,043 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,043 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,043 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,043 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,046 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,047 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,051 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,051 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,052 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,052 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,052 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:58,147 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=138.26 KB heapSize=394.23 KB 2023-03-23 19:54:58,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:58,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:58,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:58,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:58,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:58,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:54:58,152 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,156 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,158 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,158 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,160 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,160 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,160 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,160 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,162 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2018012d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,165 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,166 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,166 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,171 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,171 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601358169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601358169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601358171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601358172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601358174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,185 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,185 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601358273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601358273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,285 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,288 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,292 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,295 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73bee0fc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,298 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,298 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,302 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,302 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,302 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,310 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,310 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,410 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,413 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,415 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,417 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,417 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,417 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,417 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,418 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a2c6b3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,421 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,422 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,424 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,427 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,427 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,435 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,435 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601358476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601358477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,536 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,539 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,542 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,550 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/a89118dfe3664a3b8af61532f795cd9b 2023-03-23 19:54:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5447bd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,564 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,565 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,572 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,572 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,672 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,676 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,677 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601358676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46488 deadline: 1679601358677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601358678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,681 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba52693[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,685 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,689 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,689 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,691 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,693 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,695 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,696 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,696 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601358780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:58,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601358781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,796 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,799 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,800 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,800 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,802 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,804 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,804 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,804 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,804 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,804 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,805 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,806 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,807 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f989a1b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,807 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,808 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,810 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,812 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,812 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,821 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,821 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,821 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,922 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:58,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:58,925 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:58,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:58,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:58,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,925 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,927 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,927 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:58,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:58,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:58,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:58,928 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:58,929 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:58,931 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6b4605[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:58,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:58,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,934 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:58,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:58,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:58,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:58,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:58,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:58,938 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:58,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:58,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:58,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:58,938 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:58,939 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,939 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,940 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:58,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,941 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:58,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:58,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:58,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:58,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:58,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:58,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:58,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:58,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:58,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:58,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:58,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:58,945 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:58,945 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:58,945 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:58,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/4140023e21a447c0a6e2695346b11930 2023-03-23 19:54:59,045 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:59,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:59,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:59,048 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:59,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:59,048 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:59,049 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,049 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,050 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,050 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:59,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:59,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:59,052 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,052 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:59,052 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:59,052 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:59,052 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:59,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226b997a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:59,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:59,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,056 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:59,056 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:59,057 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,057 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:59,058 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:59,058 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,058 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,058 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:59,061 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:59,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:59,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:59,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:59,061 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:59,062 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,062 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,063 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:59,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:59,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:59,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,067 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:59,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,067 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,167 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:59,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:59,170 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:59,170 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,170 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:59,170 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:59,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,171 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,172 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,172 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:59,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:59,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,173 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:59,174 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,176 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f8a21a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:59,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:59,178 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:59,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:59,181 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:59,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:59,183 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:59,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:59,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:59,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:59,183 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:59,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,185 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:59,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:59,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,191 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:59,191 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:59,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601359283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601359287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,291 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:59,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:59,294 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:59,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:59,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:59,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,295 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,298 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:59,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:59,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:59,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:59,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:59,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:59,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:59,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:59,301 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:59,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bdbbc5c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:59,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:59,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,305 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:59,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:59,305 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:59,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:59,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:59,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:59,309 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:59,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:59,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:59,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:59,309 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:59,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,311 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:59,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:59,314 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,314 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:59,315 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:59,315 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:59,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b 2023-03-23 19:54:59,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/a89118dfe3664a3b8af61532f795cd9b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b 2023-03-23 19:54:59,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b, entries=150, sequenceid=830, filesize=11.9 K 2023-03-23 19:54:59,415 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:59,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/4140023e21a447c0a6e2695346b11930 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930 2023-03-23 19:54:59,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:59,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:59,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:59,419 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:59,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:59,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:59,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930, entries=150, sequenceid=830, filesize=11.9 K 2023-03-23 19:54:59,421 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,421 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b 2023-03-23 19:54:59,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:59,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:59,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:59,423 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,423 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:59,423 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:59,424 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:54:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:54:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:54:59,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:54:59,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b, entries=150, sequenceid=830, filesize=11.9 K 2023-03-23 19:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,427 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.26 KB/141576, heapSize ~394.18 KB/403640, currentSize=46.96 KB/48090 for 18eab3ab7700f84be1d9a1a02e88ccfa in 1280ms, sequenceid=830, compaction requested=true 2023-03-23 19:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:59,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:59,427 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:59,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:59,427 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:59,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:54:59,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a11904c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:54:59,429 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:54:59,429 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:54:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:54:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:54:59,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:54:59,431 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,431 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,431 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:54:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:54:59,432 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,432 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/34ba2c79cb9f46b896240a7bf8b3ab8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=48.5 K 2023-03-23 19:54:59,432 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,432 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:54:59,432 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,432 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/f299a59f7cdc445f845c42e5e66f7b47, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=48.5 K 2023-03-23 19:54:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:54:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,433 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 34ba2c79cb9f46b896240a7bf8b3ab8b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:59,433 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f299a59f7cdc445f845c42e5e66f7b47, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:59,433 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 3473108dc6cb418782e1b266893e8793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1679601295237 2023-03-23 19:54:59,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,433 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 686aa93b995740a9b166590da05d7ff2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1679601295237 2023-03-23 19:54:59,434 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5fa62f0474e9444f935a94ba8d944ac3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1679601296383 2023-03-23 19:54:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,434 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6dfabef5fba24f40ab5e7e3c3a517ecb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1679601296383 2023-03-23 19:54:59,434 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a89118dfe3664a3b8af61532f795cd9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:54:59,434 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4140023e21a447c0a6e2695346b11930, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:54:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:54:59,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:54:59,435 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:54:59,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:54:59,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:54:59,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:54:59,436 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:54:59,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:54:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:54:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:54:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:54:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:54:59,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,444 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:54:59,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:54:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:54:59,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:54:59,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:54:59,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:54:59,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:54:59,445 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:54:59,445 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:54:59,446 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#286 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:59,458 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/96a5c27b77a04e5b8c76f1806d5a52bf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/96a5c27b77a04e5b8c76f1806d5a52bf 2023-03-23 19:54:59,459 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/13da8e0a86e5405391d5ea6d08050f97 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/13da8e0a86e5405391d5ea6d08050f97 2023-03-23 19:54:59,467 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 96a5c27b77a04e5b8c76f1806d5a52bf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:59,467 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:59,467 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=12, startTime=1679601299427; duration=0sec 2023-03-23 19:54:59,467 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:54:59,467 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:54:59,468 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:54:59,468 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:54:59,468 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,468 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bcdcf524d77546e48992b9df3e7034fc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=48.5 K 2023-03-23 19:54:59,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bcdcf524d77546e48992b9df3e7034fc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1679601294870 2023-03-23 19:54:59,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 84f2ecba4cbf496f8a6a023925cd236f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1679601295237 2023-03-23 19:54:59,469 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 13da8e0a86e5405391d5ea6d08050f97(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:59,469 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:59,469 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=12, startTime=1679601299427; duration=0sec 2023-03-23 19:54:59,469 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:59,469 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting fd9a29194cd94928b79478b1d9f9c548, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1679601296383 2023-03-23 19:54:59,470 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f1dd5bc739a74c9cbf5bf4ac4f8b1b8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:54:59,477 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:54:59,497 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d5d61f9eeb83449dbf9c30034facfa41 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d5d61f9eeb83449dbf9c30034facfa41 2023-03-23 19:54:59,502 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into d5d61f9eeb83449dbf9c30034facfa41(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:54:59,502 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:54:59,502 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=12, startTime=1679601299427; duration=0sec 2023-03-23 19:54:59,502 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:54:59,545 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:54:59,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:54:59,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:54:59,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:54:59,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:54:59,549 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:54:59,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:54:59,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:54:59,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:54:59,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:54:59,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:54:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:54:59,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:54:59,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:54:59,553 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:54:59,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:54:59,560 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4732dc2e4fb0437c8cf258d8b552ed5a 2023-03-23 19:54:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:54:59,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:54:59,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601299698, max timestamp=1679601299707Num uniques 859; 2023-03-23 19:54:59,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:59,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601299698, max timestamp=1679601299707Num uniques 859; 2023-03-23 19:54:59,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:54:59,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679601299698, max timestamp=1679601299707Num uniques 859; 2023-03-23 19:54:59,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-23 19:54:59,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601299721, max timestamp=1679601299730Num uniques 1018; 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601299721, max timestamp=1679601299730Num uniques 1009; 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679601299721, max timestamp=1679601299730Num uniques 1018; 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-23 19:54:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601359729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-23 19:54:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601359776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601359777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601359834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601359880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:54:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601359880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:54:59,977 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/a2d74618a93b45b38888c7ba805da1f5 2023-03-23 19:54:59,993 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/2e52d063bd8749cb8e7b579c621fd8cb 2023-03-23 19:54:59,997 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4732dc2e4fb0437c8cf258d8b552ed5a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a 2023-03-23 19:55:00,000 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a, entries=150, sequenceid=843, filesize=11.9 K 2023-03-23 19:55:00,001 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/a2d74618a93b45b38888c7ba805da1f5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5 2023-03-23 19:55:00,006 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5, entries=150, sequenceid=843, filesize=11.9 K 2023-03-23 19:55:00,007 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/2e52d063bd8749cb8e7b579c621fd8cb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb 2023-03-23 19:55:00,010 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb, entries=150, sequenceid=843, filesize=11.9 K 2023-03-23 19:55:00,011 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 18eab3ab7700f84be1d9a1a02e88ccfa in 458ms, sequenceid=843, compaction requested=false 2023-03-23 19:55:00,011 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,011 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244f723d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,019 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,019 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,022 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,022 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,023 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,023 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,023 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,025 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,028 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,028 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,028 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:00,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=140.89 KB heapSize=404.77 KB 2023-03-23 19:55:00,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-23 19:55:00,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=898 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4e789bea54a74a91a2baa3b75ddb3062 2023-03-23 19:55:00,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=898 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b82358150f234805b15ea399a1ae4d11 2023-03-23 19:55:00,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601360058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601360083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601360083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,128 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,131 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,134 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,134 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,136 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,136 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,136 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,136 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,137 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c2fcdc[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,143 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,143 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,144 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,145 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,147 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,149 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,149 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,153 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,155 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,156 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,156 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,156 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601360162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,256 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,259 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,259 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,259 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,259 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,261 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57ed81e4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,272 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,276 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,276 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,277 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,277 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,277 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,278 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,284 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,284 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46510 deadline: 1679601360290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46504 deadline: 1679601360293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601360364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,384 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,387 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601360387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601360388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,393 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8f8a5[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,401 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,401 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,405 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,407 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,407 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,407 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,408 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=898 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/bf2440e32cbf4040903022e3e57a54e5 2023-03-23 19:55:00,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/4e789bea54a74a91a2baa3b75ddb3062 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062 2023-03-23 19:55:00,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062, entries=150, sequenceid=898, filesize=11.9 K 2023-03-23 19:55:00,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/b82358150f234805b15ea399a1ae4d11 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11 2023-03-23 19:55:00,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11, entries=150, sequenceid=898, filesize=11.9 K 2023-03-23 19:55:00,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/bf2440e32cbf4040903022e3e57a54e5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5 2023-03-23 19:55:00,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5, entries=150, sequenceid=898, filesize=11.9 K 2023-03-23 19:55:00,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~404.72 KB/414432, currentSize=40.25 KB/41220 for 18eab3ab7700f84be1d9a1a02e88ccfa in 448ms, sequenceid=898, compaction requested=true 2023-03-23 19:55:00,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:00,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,486 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:00,487 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:00,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,487 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:00,487 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:55:00,487 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:00,488 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,488 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:55:00,488 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/13da8e0a86e5405391d5ea6d08050f97, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.8 K 2023-03-23 19:55:00,488 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,488 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/96a5c27b77a04e5b8c76f1806d5a52bf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.8 K 2023-03-23 19:55:00,488 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 13da8e0a86e5405391d5ea6d08050f97, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:55:00,488 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 96a5c27b77a04e5b8c76f1806d5a52bf, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:55:00,488 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4732dc2e4fb0437c8cf258d8b552ed5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679601298149 2023-03-23 19:55:00,489 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a2d74618a93b45b38888c7ba805da1f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679601298149 2023-03-23 19:55:00,489 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4e789bea54a74a91a2baa3b75ddb3062, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:00,489 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b82358150f234805b15ea399a1ae4d11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:00,495 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:00,495 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#295 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:00,508 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,519 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,526 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/d9bb84907c1a469bb26ba0bf4b5aa48e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d9bb84907c1a469bb26ba0bf4b5aa48e 2023-03-23 19:55:00,530 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:00,531 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into d9bb84907c1a469bb26ba0bf4b5aa48e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,531 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:00,531 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601300486; duration=0sec 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,531 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,531 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:00,532 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:00,532 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:55:00,532 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,533 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d5d61f9eeb83449dbf9c30034facfa41, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.8 K 2023-03-23 19:55:00,533 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d5d61f9eeb83449dbf9c30034facfa41, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679601297553 2023-03-23 19:55:00,533 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2e52d063bd8749cb8e7b579c621fd8cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679601298149 2023-03-23 19:55:00,534 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bf2440e32cbf4040903022e3e57a54e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:00,538 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=908 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/95a1092846814d028a3bad0e8a1ddd9a 2023-03-23 19:55:00,540 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:00,550 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=908 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/682f60c9090d461ea146856d0982bd10 2023-03-23 19:55:00,551 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/0a1fef00a4604361a78e90988e8f0ac2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/0a1fef00a4604361a78e90988e8f0ac2 2023-03-23 19:55:00,555 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into 0a1fef00a4604361a78e90988e8f0ac2(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:00,555 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,555 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601300487; duration=0sec 2023-03-23 19:55:00,555 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:00,558 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=908 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d48163846a2f4f6dafeeacfdc7cb1a2e 2023-03-23 19:55:00,563 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/95a1092846814d028a3bad0e8a1ddd9a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a 2023-03-23 19:55:00,566 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a, entries=150, sequenceid=908, filesize=11.9 K 2023-03-23 19:55:00,567 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/682f60c9090d461ea146856d0982bd10 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10 2023-03-23 19:55:00,570 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10, entries=150, sequenceid=908, filesize=11.9 K 2023-03-23 19:55:00,570 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/d48163846a2f4f6dafeeacfdc7cb1a2e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e 2023-03-23 19:55:00,574 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e, entries=150, sequenceid=908, filesize=11.9 K 2023-03-23 19:55:00,575 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 18eab3ab7700f84be1d9a1a02e88ccfa in 45ms, sequenceid=908, compaction requested=false 2023-03-23 19:55:00,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea9bc37[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,581 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,588 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,588 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,597 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,597 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:00,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:00,698 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/8c53c57b7460409ca3c28acaad11d5be 2023-03-23 19:55:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,707 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,723 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,723 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85b9f22[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,728 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,729 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,729 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,729 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/388c60b098124587bbb39b557e91d9e9 2023-03-23 19:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,733 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,733 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,733 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,733 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,733 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,744 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,749 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,749 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9697b32c9808499787d760c4b13923e5 2023-03-23 19:55:00,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300720, max timestamp=1679601300760Num uniques 859; 2023-03-23 19:55:00,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300720, max timestamp=1679601300760Num uniques 859; 2023-03-23 19:55:00,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300720, max timestamp=1679601300760Num uniques 859; 2023-03-23 19:55:00,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/8c53c57b7460409ca3c28acaad11d5be as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be 2023-03-23 19:55:00,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be, entries=150, sequenceid=919, filesize=11.9 K 2023-03-23 19:55:00,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/388c60b098124587bbb39b557e91d9e9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9 2023-03-23 19:55:00,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9, entries=150, sequenceid=919, filesize=11.9 K 2023-03-23 19:55:00,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/9697b32c9808499787d760c4b13923e5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5 2023-03-23 19:55:00,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5, entries=150, sequenceid=919, filesize=11.9 K 2023-03-23 19:55:00,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=53.67 KB/54960 for 18eab3ab7700f84be1d9a1a02e88ccfa in 99ms, sequenceid=919, compaction requested=true 2023-03-23 19:55:00,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-23 19:55:00,780 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-23 19:55:00,781 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-23 19:55:00,781 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-23 19:55:00,781 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:00,781 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:00,783 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:00,783 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/B is initiating minor compaction (all files) 2023-03-23 19:55:00,783 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/B in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,783 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d9bb84907c1a469bb26ba0bf4b5aa48e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.9 K 2023-03-23 19:55:00,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-03-23 19:55:00,783 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d9bb84907c1a469bb26ba0bf4b5aa48e, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:00,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:00,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:00,784 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 682f60c9090d461ea146856d0982bd10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1679601300046 2023-03-23 19:55:00,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:00,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:00,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:00,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:00,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:00,784 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 388c60b098124587bbb39b557e91d9e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679601300671 2023-03-23 19:55:00,798 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#B#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:00,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/72a08394d3a44e609fe204b641bc42bd 2023-03-23 19:55:00,849 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,853 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,854 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300844, max timestamp=1679601300855Num uniques 859; 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300844, max timestamp=1679601300855Num uniques 859; 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601300844, max timestamp=1679601300855Num uniques 859; 2023-03-23 19:55:00,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,858 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,859 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,859 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,859 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,859 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,860 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3302dd6c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,862 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,863 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,866 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,866 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,871 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:00,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:00,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:00,874 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:00,874 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:00,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601360901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601360902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601360904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,926 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/328a56e372eb4b0e8b298a49202ba5ef as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/328a56e372eb4b0e8b298a49202ba5ef 2023-03-23 19:55:00,930 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 328a56e372eb4b0e8b298a49202ba5ef(size=13.1 K), total size for store is 36.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:00,930 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:00,930 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=13, startTime=1679601300486; duration=0sec 2023-03-23 19:55:00,930 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:00,931 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:00,931 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:00,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/C is initiating minor compaction (all files) 2023-03-23 19:55:00,932 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/C in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,932 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/0a1fef00a4604361a78e90988e8f0ac2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=36.9 K 2023-03-23 19:55:00,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 0a1fef00a4604361a78e90988e8f0ac2, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:00,932 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d48163846a2f4f6dafeeacfdc7cb1a2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1679601300046 2023-03-23 19:55:00,933 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9697b32c9808499787d760c4b13923e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679601300671 2023-03-23 19:55:00,940 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#C#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:00,974 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:00,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:00,978 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:00,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:00,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:00,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,979 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:00,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:00,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:00,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:00,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:00,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. started... 2023-03-23 19:55:00,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. as already flushing 2023-03-23 19:55:00,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:00,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,983 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75297679[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:00,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:00,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,987 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:00,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:00,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:00,989 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:00,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:00,991 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:00,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:00,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:00,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:00,991 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:00,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:00,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:00,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:01,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:01,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:01,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:01,000 DEBUG [Thread-1483] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:01,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:01,001 DEBUG [Thread-1483] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:01,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:01,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:01,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:01,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:01,001 DEBUG [Thread-1490] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:60301 2023-03-23 19:55:01,001 DEBUG [Thread-1490] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:01,002 DEBUG [Thread-1484] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:60301 2023-03-23 19:55:01,002 DEBUG [Thread-1484] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:01,002 DEBUG [Thread-1486] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:60301 2023-03-23 19:55:01,002 DEBUG [Thread-1486] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:01,004 DEBUG [Thread-1492] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:60301 2023-03-23 19:55:01,004 DEBUG [Thread-1492] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:01,005 DEBUG [Thread-1488] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:60301 2023-03-23 19:55:01,005 DEBUG [Thread-1488] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601361006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601361006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601361008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,101 DEBUG [Thread-1483] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:01,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601361207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601361208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601361209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,213 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/2c0528ae10d647c4a7a0fc0895793999 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2c0528ae10d647c4a7a0fc0895793999 2023-03-23 19:55:01,216 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/B of 18eab3ab7700f84be1d9a1a02e88ccfa into 2c0528ae10d647c4a7a0fc0895793999(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:01,216 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:01,216 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/B, priority=13, startTime=1679601300780; duration=0sec 2023-03-23 19:55:01,216 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:01,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/abf3e0e1f9f046dc8345616dabffdc16 2023-03-23 19:55:01,347 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/e9fa9f05be224e1a88ec31a4965c9165 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/e9fa9f05be224e1a88ec31a4965c9165 2023-03-23 19:55:01,351 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/C of 18eab3ab7700f84be1d9a1a02e88ccfa into e9fa9f05be224e1a88ec31a4965c9165(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:01,351 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:01,351 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/C, priority=13, startTime=1679601300780; duration=0sec 2023-03-23 19:55:01,351 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:01,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46528 deadline: 1679601361509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46506 deadline: 1679601361511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:01,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46488 deadline: 1679601361512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:01,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=945 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/48d567e04c384abf8446726538ea2197 2023-03-23 19:55:01,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/72a08394d3a44e609fe204b641bc42bd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd 2023-03-23 19:55:01,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd, entries=200, sequenceid=945, filesize=14.3 K 2023-03-23 19:55:01,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/abf3e0e1f9f046dc8345616dabffdc16 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/abf3e0e1f9f046dc8345616dabffdc16 2023-03-23 19:55:01,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/abf3e0e1f9f046dc8345616dabffdc16, entries=150, sequenceid=945, filesize=11.9 K 2023-03-23 19:55:01,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/48d567e04c384abf8446726538ea2197 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/48d567e04c384abf8446726538ea2197 2023-03-23 19:55:01,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/48d567e04c384abf8446726538ea2197, entries=150, sequenceid=945, filesize=11.9 K 2023-03-23 19:55:01,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 18eab3ab7700f84be1d9a1a02e88ccfa in 860ms, sequenceid=945, compaction requested=true 2023-03-23 19:55:01,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:01,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:01,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:01,643 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:01,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:01,643 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-23 19:55:01,644 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-23 19:55:01,644 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-23 19:55:01,644 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:01,644 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-23 19:55:01,644 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52376 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 18eab3ab7700f84be1d9a1a02e88ccfa/A is initiating minor compaction (all files) 2023-03-23 19:55:01,645 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 18eab3ab7700f84be1d9a1a02e88ccfa/A in TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-23 19:55:01,645 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/328a56e372eb4b0e8b298a49202ba5ef, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp, totalSize=51.1 K 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 328a56e372eb4b0e8b298a49202ba5ef, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1679601299773 2023-03-23 19:55:01,645 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 95a1092846814d028a3bad0e8a1ddd9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=908, earliestPutTs=1679601300046 2023-03-23 19:55:01,646 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 8c53c57b7460409ca3c28acaad11d5be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679601300671 2023-03-23 19:55:01,646 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 72a08394d3a44e609fe204b641bc42bd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=945, earliestPutTs=1679601300765 2023-03-23 19:55:01,652 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 18eab3ab7700f84be1d9a1a02e88ccfa#A#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:02,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:02,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-23 19:55:02,011 DEBUG [Thread-1475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:60301 2023-03-23 19:55:02,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:02,012 DEBUG [Thread-1475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:02,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:02,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:02,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:02,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:02,016 DEBUG [Thread-1477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:60301 2023-03-23 19:55:02,016 DEBUG [Thread-1477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,017 DEBUG [Thread-1473] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:60301 2023-03-23 19:55:02,017 DEBUG [Thread-1473] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,059 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/7010c31407554485b7648493936e00a9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7010c31407554485b7648493936e00a9 2023-03-23 19:55:02,063 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 18eab3ab7700f84be1d9a1a02e88ccfa/A of 18eab3ab7700f84be1d9a1a02e88ccfa into 7010c31407554485b7648493936e00a9(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:02,063 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:02,063 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa., storeName=18eab3ab7700f84be1d9a1a02e88ccfa/A, priority=12, startTime=1679601301643; duration=0sec 2023-03-23 19:55:02,063 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:02,309 DEBUG [Thread-1481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:60301 2023-03-23 19:55:02,309 DEBUG [Thread-1481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,309 DEBUG [Thread-1479] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:60301 2023-03-23 19:55:02,309 DEBUG [Thread-1479] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,309 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 297 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 282 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5096 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4763 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4818 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5092 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4734 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:55:02,310 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:55:02,310 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:60301 2023-03-23 19:55:02,310 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:02,311 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:55:02,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:55:02,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:02,315 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601302314"}]},"ts":"1679601302314"} 2023-03-23 19:55:02,316 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:55:02,318 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:55:02,318 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, UNASSIGN}] 2023-03-23 19:55:02,320 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, UNASSIGN 2023-03-23 19:55:02,320 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=18eab3ab7700f84be1d9a1a02e88ccfa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:02,320 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601302320"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601302320"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601302320"}]},"ts":"1679601302320"} 2023-03-23 19:55:02,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:02,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:02,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/1edfcd1e89f24f90988e3598b550886b 2023-03-23 19:55:02,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:02,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 18eab3ab7700f84be1d9a1a02e88ccfa, disabling compactions & flushes 2023-03-23 19:55:02,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:02,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:02,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/15f1ac2bbe604ff294c1ec1ffa7815e8 2023-03-23 19:55:02,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:03,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=985 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/da33dd9cdb8a43c78fc2ad639ed58614 2023-03-23 19:55:03,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/1edfcd1e89f24f90988e3598b550886b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1edfcd1e89f24f90988e3598b550886b 2023-03-23 19:55:03,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1edfcd1e89f24f90988e3598b550886b, entries=150, sequenceid=985, filesize=11.9 K 2023-03-23 19:55:03,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/15f1ac2bbe604ff294c1ec1ffa7815e8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/15f1ac2bbe604ff294c1ec1ffa7815e8 2023-03-23 19:55:03,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/15f1ac2bbe604ff294c1ec1ffa7815e8, entries=150, sequenceid=985, filesize=11.9 K 2023-03-23 19:55:03,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/da33dd9cdb8a43c78fc2ad639ed58614 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/da33dd9cdb8a43c78fc2ad639ed58614 2023-03-23 19:55:03,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/da33dd9cdb8a43c78fc2ad639ed58614, entries=150, sequenceid=985, filesize=11.9 K 2023-03-23 19:55:03,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=26.84 KB/27480 for 18eab3ab7700f84be1d9a1a02e88ccfa in 1242ms, sequenceid=985, compaction requested=true 2023-03-23 19:55:03,253 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:03,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:03,253 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:03,253 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. after waiting 0 ms 2023-03-23 19:55:03,253 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:03,253 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:03,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 18eab3ab7700f84be1d9a1a02e88ccfa 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-23 19:55:03,253 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. because compaction request was cancelled 2023-03-23 19:55:03,253 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=A 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=B 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 18eab3ab7700f84be1d9a1a02e88ccfa, store=C 2023-03-23 19:55:03,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:03,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:03,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=993 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/e3dfd901fe114983987cfc2d00a5e4be 2023-03-23 19:55:04,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=993 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/583e5125c7fd4982a477ad8b154a2a86 2023-03-23 19:55:04,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=993 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/6b5863ee31574b9496e9cd93e44a2900 2023-03-23 19:55:04,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/A/e3dfd901fe114983987cfc2d00a5e4be as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e3dfd901fe114983987cfc2d00a5e4be 2023-03-23 19:55:04,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e3dfd901fe114983987cfc2d00a5e4be, entries=100, sequenceid=993, filesize=9.5 K 2023-03-23 19:55:04,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/B/583e5125c7fd4982a477ad8b154a2a86 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/583e5125c7fd4982a477ad8b154a2a86 2023-03-23 19:55:04,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/583e5125c7fd4982a477ad8b154a2a86, entries=100, sequenceid=993, filesize=9.5 K 2023-03-23 19:55:04,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/.tmp/C/6b5863ee31574b9496e9cd93e44a2900 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/6b5863ee31574b9496e9cd93e44a2900 2023-03-23 19:55:04,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/6b5863ee31574b9496e9cd93e44a2900, entries=100, sequenceid=993, filesize=9.5 K 2023-03-23 19:55:04,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 18eab3ab7700f84be1d9a1a02e88ccfa in 839ms, sequenceid=993, compaction requested=true 2023-03-23 19:55:04,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6c818906d8ce4beba73b2818c1f1bc84, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/81dac31617b64d3281b98e7175353ee2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e59dd7acd2bf4952b9d781b849f6dbd3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e2c6a16b17b54a8abc16da1853ad3955, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8ccf379d30804874a5c1df3b39b034b8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/67bd2a5033714da4804799352f7b7e73, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/34ba2c79cb9f46b896240a7bf8b3ab8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/13da8e0a86e5405391d5ea6d08050f97, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/328a56e372eb4b0e8b298a49202ba5ef, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd] to archive 2023-03-23 19:55:04,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:04,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7cc4bb4797304b82a1012ead5cda8e4b 2023-03-23 19:55:04,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4fe85194410d47629e30d2f9ed2f8417 2023-03-23 19:55:04,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1b44852599434d578d756597f62fc4d4 2023-03-23 19:55:04,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6c818906d8ce4beba73b2818c1f1bc84 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6c818906d8ce4beba73b2818c1f1bc84 2023-03-23 19:55:04,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/601245484906478990c1f26c78dddaed 2023-03-23 19:55:04,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/960af9037afd440581f5cbe382f08d7b 2023-03-23 19:55:04,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/81dac31617b64d3281b98e7175353ee2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/81dac31617b64d3281b98e7175353ee2 2023-03-23 19:55:04,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/59d1a9cad0824a4982482cd8ccb45e72 2023-03-23 19:55:04,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/0162beca5dbf4b33aa67952cec3e6711 2023-03-23 19:55:04,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e59dd7acd2bf4952b9d781b849f6dbd3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e59dd7acd2bf4952b9d781b849f6dbd3 2023-03-23 19:55:04,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4f80c8897f6c43b38ae1e35d047dbdd2 2023-03-23 19:55:04,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/656e8568ce9c4e8fafe47244771f47e0 2023-03-23 19:55:04,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/9ccb1d4634b0425786253a48f2e6e320 2023-03-23 19:55:04,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e2c6a16b17b54a8abc16da1853ad3955 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e2c6a16b17b54a8abc16da1853ad3955 2023-03-23 19:55:04,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/66edee93c1aa4b2b883ac35f48bdf7ec 2023-03-23 19:55:04,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/65d506d2d1a041ddbc069b352582ae25 2023-03-23 19:55:04,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/6187ab99f2c24ed29fb283724257bca2 2023-03-23 19:55:04,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/d81470317abe47c0afa06dec11bfa265 2023-03-23 19:55:04,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/597d6fe48159471bbce18b61d006ccd6 2023-03-23 19:55:04,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/87af869eac384364b650f3f78ce25f0b 2023-03-23 19:55:04,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3bbd9bd7fde3491ea3f9e12bf2994ffb 2023-03-23 19:55:04,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/cec50c940f6744b9811c95ac6b4193cd 2023-03-23 19:55:04,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8ccf379d30804874a5c1df3b39b034b8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8ccf379d30804874a5c1df3b39b034b8 2023-03-23 19:55:04,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/035d0b10ac714f87b8c9aceda9272307 2023-03-23 19:55:04,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/67bd2a5033714da4804799352f7b7e73 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/67bd2a5033714da4804799352f7b7e73 2023-03-23 19:55:04,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4006dd92e6084fa1b97eb269a70d320e 2023-03-23 19:55:04,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/53b4040df2db4ff0be00004faff45e6d 2023-03-23 19:55:04,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/34ba2c79cb9f46b896240a7bf8b3ab8b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/34ba2c79cb9f46b896240a7bf8b3ab8b 2023-03-23 19:55:04,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/aa205c53abba4d8d816277551562733a 2023-03-23 19:55:04,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/3473108dc6cb418782e1b266893e8793 2023-03-23 19:55:04,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/5fa62f0474e9444f935a94ba8d944ac3 2023-03-23 19:55:04,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/13da8e0a86e5405391d5ea6d08050f97 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/13da8e0a86e5405391d5ea6d08050f97 2023-03-23 19:55:04,149 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/a89118dfe3664a3b8af61532f795cd9b 2023-03-23 19:55:04,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4732dc2e4fb0437c8cf258d8b552ed5a 2023-03-23 19:55:04,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/328a56e372eb4b0e8b298a49202ba5ef to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/328a56e372eb4b0e8b298a49202ba5ef 2023-03-23 19:55:04,152 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/4e789bea54a74a91a2baa3b75ddb3062 2023-03-23 19:55:04,153 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/95a1092846814d028a3bad0e8a1ddd9a 2023-03-23 19:55:04,154 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/8c53c57b7460409ca3c28acaad11d5be 2023-03-23 19:55:04,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/72a08394d3a44e609fe204b641bc42bd 2023-03-23 19:55:04,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/42128d080f0d4825ac1f0781effb0246, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/18d8e64fd4524e55b689607b00ea1187, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/76c2c6a06df040d0b39b63ccbedcdc32, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/792ba1ad96164fda9d2458bf8ae3191d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b64c74f2bdb44a398bf8dfefc978e19d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d5876672a5e46759ccc4eb1001cb95c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/f299a59f7cdc445f845c42e5e66f7b47, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/96a5c27b77a04e5b8c76f1806d5a52bf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d9bb84907c1a469bb26ba0bf4b5aa48e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9] to archive 2023-03-23 19:55:04,159 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:04,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6a8a2ab33763459eb7e2cfa955ee77fe 2023-03-23 19:55:04,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2f76c9b19b3344edb009e491dffe535e 2023-03-23 19:55:04,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/42128d080f0d4825ac1f0781effb0246 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/42128d080f0d4825ac1f0781effb0246 2023-03-23 19:55:04,163 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/ff4f38f890f14c73939191dd73b2a822 2023-03-23 19:55:04,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d7266bc46e541c6a87d8e1823cded28 2023-03-23 19:55:04,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/18d8e64fd4524e55b689607b00ea1187 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/18d8e64fd4524e55b689607b00ea1187 2023-03-23 19:55:04,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c9d34a409c5d453288f4f46c480caa10 2023-03-23 19:55:04,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/81548c5352f84267a053cec7965ed7db 2023-03-23 19:55:04,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/76c2c6a06df040d0b39b63ccbedcdc32 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/76c2c6a06df040d0b39b63ccbedcdc32 2023-03-23 19:55:04,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d59c31a0d9fd4776bdf0557a5e48e266 2023-03-23 19:55:04,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3a3de4088c274a989e07cdcf9c6b9fb6 2023-03-23 19:55:04,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a6451500a43840ff98ebaf24c0428b0e 2023-03-23 19:55:04,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/792ba1ad96164fda9d2458bf8ae3191d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/792ba1ad96164fda9d2458bf8ae3191d 2023-03-23 19:55:04,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/24a6cbdf969740d393403be0e85dcd8a 2023-03-23 19:55:04,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8c0eeccd17764faaaf016a642e755065 2023-03-23 19:55:04,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/c6f5ff9cb80c4d64a761e8618dc99cf9 2023-03-23 19:55:04,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b25cccb7584747078d940be4391af6dc 2023-03-23 19:55:04,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/86e0ae8682f64dd286d65d5e4b09d878 2023-03-23 19:55:04,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/3e0ae19323db44009794fa9140047d7b 2023-03-23 19:55:04,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2a39e0c87571491d8da7266b560a0e51 2023-03-23 19:55:04,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/1ddc6acd65b64f368bfd8f802a698afb 2023-03-23 19:55:04,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b64c74f2bdb44a398bf8dfefc978e19d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b64c74f2bdb44a398bf8dfefc978e19d 2023-03-23 19:55:04,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/bc14e526c859417581a170d4933a8b1c 2023-03-23 19:55:04,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/5d95122a741e46598704e8d58e6be1f1 2023-03-23 19:55:04,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d5876672a5e46759ccc4eb1001cb95c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/9d5876672a5e46759ccc4eb1001cb95c 2023-03-23 19:55:04,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b09cdf5dfed543468d58704a9e1583e5 2023-03-23 19:55:04,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/8558de91c5fb4398876fee64a3ce8029 2023-03-23 19:55:04,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/f299a59f7cdc445f845c42e5e66f7b47 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/f299a59f7cdc445f845c42e5e66f7b47 2023-03-23 19:55:04,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4ed9b0d451ce47098c28418d6742854a 2023-03-23 19:55:04,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/686aa93b995740a9b166590da05d7ff2 2023-03-23 19:55:04,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/6dfabef5fba24f40ab5e7e3c3a517ecb 2023-03-23 19:55:04,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/96a5c27b77a04e5b8c76f1806d5a52bf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/96a5c27b77a04e5b8c76f1806d5a52bf 2023-03-23 19:55:04,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/4140023e21a447c0a6e2695346b11930 2023-03-23 19:55:04,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/a2d74618a93b45b38888c7ba805da1f5 2023-03-23 19:55:04,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d9bb84907c1a469bb26ba0bf4b5aa48e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/d9bb84907c1a469bb26ba0bf4b5aa48e 2023-03-23 19:55:04,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/b82358150f234805b15ea399a1ae4d11 2023-03-23 19:55:04,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/682f60c9090d461ea146856d0982bd10 2023-03-23 19:55:04,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/388c60b098124587bbb39b557e91d9e9 2023-03-23 19:55:04,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/657a77d45f4b49be9ca350655a98fe8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/3df710b570f04566a71889c0e2308513, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/80648660a012498ca14185876c4d5da9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ab7e882597074f679ecfcfba336421a7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5b7ce4811e4046f0a58b09a38df7e577, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/65a4c046f56f449a91826db601a77469, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bcdcf524d77546e48992b9df3e7034fc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d5d61f9eeb83449dbf9c30034facfa41, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/0a1fef00a4604361a78e90988e8f0ac2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5] to archive 2023-03-23 19:55:04,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:04,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9c13183b938f49819e6d3c7725ea533a 2023-03-23 19:55:04,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c7fbc02d72394e02b7efd1a447273484 2023-03-23 19:55:04,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/657a77d45f4b49be9ca350655a98fe8b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/657a77d45f4b49be9ca350655a98fe8b 2023-03-23 19:55:04,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a34787e59cd847c8896fd6f7c2faeb18 2023-03-23 19:55:04,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7d817cfa025b403a93f064698c434ebd 2023-03-23 19:55:04,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/3df710b570f04566a71889c0e2308513 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/3df710b570f04566a71889c0e2308513 2023-03-23 19:55:04,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c84b22c5511242a1913d8a8c56211327 2023-03-23 19:55:04,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a553bd830fe94b7c9b6f01b88e47b187 2023-03-23 19:55:04,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/80648660a012498ca14185876c4d5da9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/80648660a012498ca14185876c4d5da9 2023-03-23 19:55:04,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9606b5457c0a4f5a8484215e02438d8d 2023-03-23 19:55:04,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fdd49e9339b549f6bb139fd5dd4fc1dd 2023-03-23 19:55:04,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d95ce270c47343b19b4386891b95829d 2023-03-23 19:55:04,209 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ab7e882597074f679ecfcfba336421a7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ab7e882597074f679ecfcfba336421a7 2023-03-23 19:55:04,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/35d8060e4e554c37bbd00fb63108ba46 2023-03-23 19:55:04,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fc5c55602a2146448899ac43be595717 2023-03-23 19:55:04,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c331aa783bad440c8346f01568465a23 2023-03-23 19:55:04,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d264b94158ce43efb1df7cbd75a0bf85 2023-03-23 19:55:04,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/7ece86f5263748138ca0bf305ffb7427 2023-03-23 19:55:04,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/ca076b452ee048c4b82ff081165632d0 2023-03-23 19:55:04,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/8bbe9447a7b24d53ac2cfbaae8a2c0b5 2023-03-23 19:55:04,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/1f3467321cb0472c9f2d89b3cb62dee3 2023-03-23 19:55:04,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5b7ce4811e4046f0a58b09a38df7e577 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5b7ce4811e4046f0a58b09a38df7e577 2023-03-23 19:55:04,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/c0d31060fa5a41ce99ea9758f2428192 2023-03-23 19:55:04,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/447f9fd8c14e4166b234fb185fd1dd4d 2023-03-23 19:55:04,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/65a4c046f56f449a91826db601a77469 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/65a4c046f56f449a91826db601a77469 2023-03-23 19:55:04,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/a6bb7457c6ad411db3dac342a850523d 2023-03-23 19:55:04,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/5620f25acd954c63a7267c74d5d8f2d2 2023-03-23 19:55:04,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bcdcf524d77546e48992b9df3e7034fc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bcdcf524d77546e48992b9df3e7034fc 2023-03-23 19:55:04,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/eb474f64f2964e82834d5b063e8bf7ce 2023-03-23 19:55:04,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/84f2ecba4cbf496f8a6a023925cd236f 2023-03-23 19:55:04,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/fd9a29194cd94928b79478b1d9f9c548 2023-03-23 19:55:04,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d5d61f9eeb83449dbf9c30034facfa41 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d5d61f9eeb83449dbf9c30034facfa41 2023-03-23 19:55:04,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/f1dd5bc739a74c9cbf5bf4ac4f8b1b8b 2023-03-23 19:55:04,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/2e52d063bd8749cb8e7b579c621fd8cb 2023-03-23 19:55:04,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/0a1fef00a4604361a78e90988e8f0ac2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/0a1fef00a4604361a78e90988e8f0ac2 2023-03-23 19:55:04,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/bf2440e32cbf4040903022e3e57a54e5 2023-03-23 19:55:04,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/d48163846a2f4f6dafeeacfdc7cb1a2e 2023-03-23 19:55:04,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/9697b32c9808499787d760c4b13923e5 2023-03-23 19:55:04,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/recovered.edits/996.seqid, newMaxSeqId=996, maxSeqId=1 2023-03-23 19:55:04,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa. 2023-03-23 19:55:04,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 18eab3ab7700f84be1d9a1a02e88ccfa: 2023-03-23 19:55:04,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:04,235 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=18eab3ab7700f84be1d9a1a02e88ccfa, regionState=CLOSED 2023-03-23 19:55:04,235 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601304235"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601304235"}]},"ts":"1679601304235"} 2023-03-23 19:55:04,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-23 19:55:04,237 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 18eab3ab7700f84be1d9a1a02e88ccfa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 1.9150 sec 2023-03-23 19:55:04,238 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-23 19:55:04,238 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=18eab3ab7700f84be1d9a1a02e88ccfa, UNASSIGN in 1.9190 sec 2023-03-23 19:55:04,239 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601304239"}]},"ts":"1679601304239"} 2023-03-23 19:55:04,240 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:55:04,241 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:55:04,242 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9310 sec 2023-03-23 19:55:04,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-23 19:55:04,419 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-23 19:55:04,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:55:04,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,421 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-23 19:55:04,422 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,426 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:04,428 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/recovered.edits] 2023-03-23 19:55:04,430 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1edfcd1e89f24f90988e3598b550886b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/1edfcd1e89f24f90988e3598b550886b 2023-03-23 19:55:04,431 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7010c31407554485b7648493936e00a9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/7010c31407554485b7648493936e00a9 2023-03-23 19:55:04,432 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e3dfd901fe114983987cfc2d00a5e4be to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/A/e3dfd901fe114983987cfc2d00a5e4be 2023-03-23 19:55:04,434 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/15f1ac2bbe604ff294c1ec1ffa7815e8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/15f1ac2bbe604ff294c1ec1ffa7815e8 2023-03-23 19:55:04,435 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2c0528ae10d647c4a7a0fc0895793999 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/2c0528ae10d647c4a7a0fc0895793999 2023-03-23 19:55:04,436 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/583e5125c7fd4982a477ad8b154a2a86 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/583e5125c7fd4982a477ad8b154a2a86 2023-03-23 19:55:04,438 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/abf3e0e1f9f046dc8345616dabffdc16 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/B/abf3e0e1f9f046dc8345616dabffdc16 2023-03-23 19:55:04,439 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/48d567e04c384abf8446726538ea2197 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/48d567e04c384abf8446726538ea2197 2023-03-23 19:55:04,440 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/6b5863ee31574b9496e9cd93e44a2900 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/6b5863ee31574b9496e9cd93e44a2900 2023-03-23 19:55:04,441 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/da33dd9cdb8a43c78fc2ad639ed58614 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/da33dd9cdb8a43c78fc2ad639ed58614 2023-03-23 19:55:04,442 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/e9fa9f05be224e1a88ec31a4965c9165 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/C/e9fa9f05be224e1a88ec31a4965c9165 2023-03-23 19:55:04,444 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/recovered.edits/996.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa/recovered.edits/996.seqid 2023-03-23 19:55:04,444 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/18eab3ab7700f84be1d9a1a02e88ccfa 2023-03-23 19:55:04,444 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:55:04,446 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,447 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:55:04,449 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:55:04,449 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,450 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:55:04,450 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601304450"}]},"ts":"9223372036854775807"} 2023-03-23 19:55:04,451 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:55:04,451 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 18eab3ab7700f84be1d9a1a02e88ccfa, NAME => 'TestAcidGuarantees,,1679601278825.18eab3ab7700f84be1d9a1a02e88ccfa.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:55:04,451 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:55:04,451 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601304451"}]},"ts":"9223372036854775807"} 2023-03-23 19:55:04,452 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:55:04,454 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,454 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-03-23 19:55:04,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-23 19:55:04,523 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-23 19:55:04,534 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=232 (was 233), OpenFileDescriptor=568 (was 542) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=368 (was 377), ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=5394 (was 5466) 2023-03-23 19:55:04,546 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=232, OpenFileDescriptor=568, MaxFileDescriptor=60000, SystemLoadAverage=368, ProcessCount=171, AvailableMemoryMB=5394 2023-03-23 19:55:04,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:55:04,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:55:04,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:04,550 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:55:04,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-23 19:55:04,551 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:55:04,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:04,552 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:04,552 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa empty. 2023-03-23 19:55:04,553 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:04,553 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:55:04,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:04,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:04,959 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:55:04,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 12d8d2c95ad908a17b68eef8e95649aa, NAME => 'TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:55:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 12d8d2c95ad908a17b68eef8e95649aa, disabling compactions & flushes 2023-03-23 19:55:05,367 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. after waiting 0 ms 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,367 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,367 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:05,369 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:55:05,370 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601305370"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601305370"}]},"ts":"1679601305370"} 2023-03-23 19:55:05,371 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:55:05,371 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:55:05,371 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601305371"}]},"ts":"1679601305371"} 2023-03-23 19:55:05,372 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:55:05,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, ASSIGN}] 2023-03-23 19:55:05,377 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, ASSIGN 2023-03-23 19:55:05,377 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:55:05,528 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:05,529 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601305528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601305528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601305528"}]},"ts":"1679601305528"} 2023-03-23 19:55:05,530 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:05,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:05,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 12d8d2c95ad908a17b68eef8e95649aa, NAME => 'TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:55:05,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:55:05,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,686 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,687 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A 2023-03-23 19:55:05,687 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A 2023-03-23 19:55:05,687 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:05,687 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName A 2023-03-23 19:55:05,688 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:05,688 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,689 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B 2023-03-23 19:55:05,689 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B 2023-03-23 19:55:05,689 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:05,689 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName B 2023-03-23 19:55:05,689 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:05,689 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,690 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C 2023-03-23 19:55:05,690 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C 2023-03-23 19:55:05,690 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:05,691 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName C 2023-03-23 19:55:05,691 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:05,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:55:05,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:05,695 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:55:05,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 12d8d2c95ad908a17b68eef8e95649aa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63561290, jitterRate=-0.052862972021102905}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:55:05,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:05,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., pid=37, masterSystemTime=1679601305681 2023-03-23 19:55:05,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:05,698 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:05,698 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601305698"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601305698"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601305698"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601305698"}]},"ts":"1679601305698"} 2023-03-23 19:55:05,700 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-23 19:55:05,700 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 169 msec 2023-03-23 19:55:05,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-23 19:55:05,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, ASSIGN in 325 msec 2023-03-23 19:55:05,701 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:55:05,701 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601305701"}]},"ts":"1679601305701"} 2023-03-23 19:55:05,702 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:55:05,704 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:55:05,705 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-03-23 19:55:06,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-23 19:55:06,655 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-23 19:55:06,656 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:06,661 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:06,662 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:06,664 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57058, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:06,665 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:55:06,666 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48674, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:55:06,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:55:06,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:55:06,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:07,078 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:55:07,079 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:55:07,079 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:55:07,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-23 19:55:07,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, REOPEN/MOVE}] 2023-03-23 19:55:07,084 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, REOPEN/MOVE 2023-03-23 19:55:07,084 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,085 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601307084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601307084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601307084"}]},"ts":"1679601307084"} 2023-03-23 19:55:07,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:07,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 12d8d2c95ad908a17b68eef8e95649aa, disabling compactions & flushes 2023-03-23 19:55:07,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. after waiting 0 ms 2023-03-23 19:55:07,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-23 19:55:07,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:07,242 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 12d8d2c95ad908a17b68eef8e95649aa to self. 2023-03-23 19:55:07,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,243 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=CLOSED 2023-03-23 19:55:07,243 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601307243"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601307243"}]},"ts":"1679601307243"} 2023-03-23 19:55:07,245 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-23 19:55:07,245 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 159 msec 2023-03-23 19:55:07,245 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=true 2023-03-23 19:55:07,396 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,396 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601307396"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601307396"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601307396"}]},"ts":"1679601307396"} 2023-03-23 19:55:07,397 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:07,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 12d8d2c95ad908a17b68eef8e95649aa, NAME => 'TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:55:07,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:55:07,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,553 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,554 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A 2023-03-23 19:55:07,554 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A 2023-03-23 19:55:07,555 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:07,555 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName A 2023-03-23 19:55:07,556 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:07,556 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,557 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B 2023-03-23 19:55:07,557 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B 2023-03-23 19:55:07,557 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:07,557 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName B 2023-03-23 19:55:07,558 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:07,558 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,558 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C 2023-03-23 19:55:07,558 DEBUG [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C 2023-03-23 19:55:07,559 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:07,559 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12d8d2c95ad908a17b68eef8e95649aa columnFamilyName C 2023-03-23 19:55:07,559 INFO [StoreOpener-12d8d2c95ad908a17b68eef8e95649aa-1] regionserver.HStore(310): Store=12d8d2c95ad908a17b68eef8e95649aa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:07,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:55:07,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 12d8d2c95ad908a17b68eef8e95649aa; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70265628, jitterRate=0.0470394492149353}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:55:07,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:07,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., pid=42, masterSystemTime=1679601307549 2023-03-23 19:55:07,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,567 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,567 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601307567"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601307567"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601307567"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601307567"}]},"ts":"1679601307567"} 2023-03-23 19:55:07,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-23 19:55:07,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 171 msec 2023-03-23 19:55:07,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-23 19:55:07,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, REOPEN/MOVE in 486 msec 2023-03-23 19:55:07,571 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-23 19:55:07,571 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-23 19:55:07,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-03-23 19:55:07,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-23 19:55:07,574 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,579 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,579 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,583 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,583 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,587 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,587 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,594 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,594 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,601 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,601 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,605 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,606 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,611 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,611 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,621 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,621 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x159c5e0c to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,624 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1774338b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,625 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec116b to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:07,632 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cf75c24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:07,637 DEBUG [hconnection-0x38b6acb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,637 DEBUG [hconnection-0x7363d647-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,638 DEBUG [hconnection-0x19681f58-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,638 DEBUG [hconnection-0xdefc48a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,639 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,644 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57088, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,644 DEBUG [hconnection-0x504a11a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,644 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57086, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,644 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,645 DEBUG [hconnection-0x57d3e671-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,645 DEBUG [hconnection-0x705fc06f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,645 DEBUG [hconnection-0x65599ce2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,649 DEBUG [hconnection-0x2f204606-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,650 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57104, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,651 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57102, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,653 DEBUG [hconnection-0x757ea819-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:07,654 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:07,656 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,658 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,660 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:07,671 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:07,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:07,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:07,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:07,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:07,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,677 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:07,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:07,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:07,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:07,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:07,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:07,679 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:07,682 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,684 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f224ef[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:07,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:07,690 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:07,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:07,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:07,697 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:07,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:07,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:07,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:07,697 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:07,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601367693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:07,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601367696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:07,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601367697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601367698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601367699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:07,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:07,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:07,705 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:07,705 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:07,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:07,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,705 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,705 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601367799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601367800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601367801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601367801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601367801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,805 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:07,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:07,812 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:07,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:07,813 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,813 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,814 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:07,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:07,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:07,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:07,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:07,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:07,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:07,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:07,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:07,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:07,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:07,825 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3284059c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:07,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:07,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,828 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:07,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:07,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:07,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:07,833 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:07,836 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:07,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:07,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:07,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:07,836 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:07,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:07,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,840 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,842 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:07,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:07,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:07,842 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:07,842 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:07,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:07,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:07,852 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,943 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:07,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:07,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:07,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:07,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:07,946 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:07,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:07,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:07,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,949 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:07,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:07,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:07,952 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,952 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:07,952 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:07,952 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:07,954 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c8471d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:07,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:07,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:07,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:07,958 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,958 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:07,961 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:07,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:07,962 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:07,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:07,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:07,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,966 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:07,968 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:07,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:07,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:07,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:07,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:07,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:07,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:07,969 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:07,969 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:07,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:07,969 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601368004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601368004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601368005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601368006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601368006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,069 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,072 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,074 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,076 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,076 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,076 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,076 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,077 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ec6123[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,079 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,079 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,085 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,088 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,088 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,090 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,090 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,090 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,094 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,095 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,095 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,116 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032386ab0864223d4fc0ac9f0e42f1b2e1f6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032386ab0864223d4fc0ac9f0e42f1b2e1f6 2023-03-23 19:55:08,121 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=19, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/22e6de6cc05449418578fd6dfa01e0b2 2023-03-23 19:55:08,195 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,198 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,199 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,199 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,203 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,203 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,203 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,203 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,205 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecefdad[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,209 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,211 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,212 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,214 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,216 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,222 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,223 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,223 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,223 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601368310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601368310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601368311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601368311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601368312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,323 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,326 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,326 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,326 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,326 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,328 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cf4bb0d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,341 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,341 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,341 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,347 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,347 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,448 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,457 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,458 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1951fd21[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,469 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,469 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,473 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,473 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,473 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,474 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,477 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,477 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,477 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/e76bfab757ad416d94a824d853b6d84d 2023-03-23 19:55:08,578 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,581 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537e7251[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,590 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,604 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,604 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,705 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,711 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,713 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,713 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,713 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a74ce06[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,723 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,723 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,723 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,723 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,727 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,727 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,728 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,729 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,729 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,733 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,733 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,733 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,733 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:08,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601368815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601368816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601368816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601368818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601368819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,833 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,837 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,840 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,840 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,840 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:08,840 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:08,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13313e96[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:08,844 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:08,844 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,845 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:08,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:08,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:08,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:08,847 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:08,849 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:08,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:08,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:08,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:08,849 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:08,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:08,851 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:08,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,851 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:08,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:08,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:08,856 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:08,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:08,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,856 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:08,856 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:08,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:08,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/2d5fade8c08f4a8aafe7076db92b649a 2023-03-23 19:55:08,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/22e6de6cc05449418578fd6dfa01e0b2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2 2023-03-23 19:55:08,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2, entries=250, sequenceid=19, filesize=38.7 K 2023-03-23 19:55:08,956 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:08,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/e76bfab757ad416d94a824d853b6d84d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d 2023-03-23 19:55:08,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:08,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d, entries=150, sequenceid=19, filesize=11.6 K 2023-03-23 19:55:08,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/2d5fade8c08f4a8aafe7076db92b649a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a 2023-03-23 19:55:08,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:08,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:08,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:08,965 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:08,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:08,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:08,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,966 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,967 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:08,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:08,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:08,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a, entries=150, sequenceid=19, filesize=11.6 K 2023-03-23 19:55:08,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:08,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 12d8d2c95ad908a17b68eef8e95649aa in 1293ms, sequenceid=19, compaction requested=false 2023-03-23 19:55:08,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:08,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:08,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:08,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:08,975 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:08,975 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:08,975 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:08,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:08,977 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:08,977 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:08,977 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:09,394 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323ac4505883cc542b3be7e71a1c94475b0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ac4505883cc542b3be7e71a1c94475b0 2023-03-23 19:55:09,773 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:55:09,800 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/eaff0b0e86174350bc79295cdbf6a2b6 2023-03-23 19:55:09,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:09,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:09,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601369837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601369838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601369841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601369842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601369842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601369943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601369946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601369948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601369948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:09,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601369948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601370151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601370152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601370152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601370153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601370153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,217 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/b2ab7d9e7cab4fcca6cdccce115bfb75 2023-03-23 19:55:10,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601370457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601370457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601370458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601370459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601370459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,629 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/774497d3915d4d2086e4c31052bb287e 2023-03-23 19:55:10,632 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/eaff0b0e86174350bc79295cdbf6a2b6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6 2023-03-23 19:55:10,635 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6, entries=150, sequenceid=37, filesize=25.1 K 2023-03-23 19:55:10,636 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/b2ab7d9e7cab4fcca6cdccce115bfb75 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75 2023-03-23 19:55:10,640 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75, entries=150, sequenceid=37, filesize=11.6 K 2023-03-23 19:55:10,640 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/774497d3915d4d2086e4c31052bb287e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e 2023-03-23 19:55:10,643 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e, entries=150, sequenceid=37, filesize=11.6 K 2023-03-23 19:55:10,644 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 12d8d2c95ad908a17b68eef8e95649aa in 1667ms, sequenceid=37, compaction requested=false 2023-03-23 19:55:10,644 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:10,644 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:10,644 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:10,644 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:10,644 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,644 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:10,644 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:10,646 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,646 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:10,646 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,646 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:10,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:10,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f7124[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:10,648 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:10,648 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,649 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:10,649 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:10,649 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:10,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:10,651 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,651 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:10,651 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:10,652 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:10,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:10,654 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:10,654 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:10,654 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:10,654 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:10,654 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:10,656 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,656 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:10,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,656 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:10,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,657 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,659 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,659 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:10,659 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:10,659 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:10,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:10,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:10,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:10,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:10,660 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:10,660 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:10,659 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:10,660 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:10,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:10,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:10,660 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,760 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:10,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:10,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:10,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:10,763 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:10,763 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:10,763 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:10,763 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:10,764 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,764 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:10,769 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,769 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:10,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:10,769 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,769 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:10,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:10,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:10,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:10,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:10,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:10,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:10,770 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:10,771 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:10,771 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:10,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:10,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601370981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601370982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601370982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601370983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:10,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:10,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601370984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601371087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601371089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601371089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601371089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601371090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,184 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323caa8eaabd1c943c8844bc7ced1e611d7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323caa8eaabd1c943c8844bc7ced1e611d7 2023-03-23 19:55:11,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601371292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601371294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601371294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601371295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601371295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,590 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/9884a4b784204d1195b74a425e53f4f4 2023-03-23 19:55:11,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601371597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601371598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,603 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/c29b5f07634942eba3650f9dab3ccaf6 2023-03-23 19:55:11,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601371600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601371600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:11,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:11,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601371600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,013 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/fc7ae564b92242a7a31b9224c143c98d 2023-03-23 19:55:12,017 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/9884a4b784204d1195b74a425e53f4f4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4 2023-03-23 19:55:12,020 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4, entries=150, sequenceid=51, filesize=25.1 K 2023-03-23 19:55:12,021 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/c29b5f07634942eba3650f9dab3ccaf6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6 2023-03-23 19:55:12,024 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6, entries=150, sequenceid=51, filesize=11.6 K 2023-03-23 19:55:12,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/fc7ae564b92242a7a31b9224c143c98d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d 2023-03-23 19:55:12,028 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d, entries=150, sequenceid=51, filesize=11.6 K 2023-03-23 19:55:12,029 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 12d8d2c95ad908a17b68eef8e95649aa in 1258ms, sequenceid=51, compaction requested=true 2023-03-23 19:55:12,029 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:12,029 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:12,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:12,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,031 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25d3673c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:12,032 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:12,032 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,033 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,034 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:12,034 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:12,034 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:12,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,035 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,035 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:12,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:12,037 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:12,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:12,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:12,037 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:12,037 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:12,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,040 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,041 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,041 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,042 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:12,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:12,045 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,045 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:12,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,116 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323d8f046e34c1640e7be93ecf341165953 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d8f046e34c1640e7be93ecf341165953 2023-03-23 19:55:12,122 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/a22379685cbc41228c554482bc0ad2cf 2023-03-23 19:55:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601372114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601372116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601372121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/ce50a23b25aa446f8e9c116d2088e9fa 2023-03-23 19:55:12,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601372125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601372125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,145 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:12,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:12,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:12,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:12,149 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:12,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:12,149 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:12,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,151 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,152 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:12,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:12,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:12,153 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,153 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:12,153 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:12,153 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:12,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,155 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,155 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790b897e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:12,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:12,156 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,158 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:12,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,159 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:12,160 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:12,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,160 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:12,162 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:12,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:12,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:12,162 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:12,162 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:12,164 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,164 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,164 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,164 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,166 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,168 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,168 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:12,169 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,169 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:12,169 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:12,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601372226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601372226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601372226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601372238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601372238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,269 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:12,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:12,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:12,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:12,272 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:12,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:12,272 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:12,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,274 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,275 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,275 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:12,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:12,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:12,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:12,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:12,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:12,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6354cd47[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:12,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:12,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,281 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:12,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:12,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:12,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:12,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:12,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:12,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:12,290 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:12,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:12,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:12,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:12,290 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:12,292 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,292 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,292 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,294 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,295 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:12,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,297 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:12,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,297 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:12,398 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:12,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:12,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:12,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:12,412 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:12,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:12,412 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:12,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,413 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,415 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,415 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:12,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:12,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:12,420 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,420 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:12,420 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:12,420 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:12,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:12,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dbc9658[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:12,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:12,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,432 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:12,432 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:12,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:12,438 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:12,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:12,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:12,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:12,438 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601372433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601372434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601372434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:12,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:12,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,445 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:12,445 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601372442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601372443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,545 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:12,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:12,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/58a95273aad440ba96f62a996004a87e 2023-03-23 19:55:12,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:12,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/a22379685cbc41228c554482bc0ad2cf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf 2023-03-23 19:55:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:12,553 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:12,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:12,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:12,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf, entries=200, sequenceid=70, filesize=31.9 K 2023-03-23 19:55:12,555 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,555 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/ce50a23b25aa446f8e9c116d2088e9fa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa 2023-03-23 19:55:12,555 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:12,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:12,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:12,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:12,557 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,557 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:12,557 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:12,558 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:12,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2368f854[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:12,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:12,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa, entries=150, sequenceid=70, filesize=11.6 K 2023-03-23 19:55:12,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/58a95273aad440ba96f62a996004a87e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e 2023-03-23 19:55:12,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:12,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:12,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:12,565 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e, entries=150, sequenceid=70, filesize=11.6 K 2023-03-23 19:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 12d8d2c95ad908a17b68eef8e95649aa in 463ms, sequenceid=70, compaction requested=true 2023-03-23 19:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:12,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:12,566 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:12,566 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:12,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,568 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123808 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:12,568 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 12d8d2c95ad908a17b68eef8e95649aa/A is initiating minor compaction (all files) 2023-03-23 19:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,568 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12d8d2c95ad908a17b68eef8e95649aa/A in TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,568 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp, totalSize=120.9 K 2023-03-23 19:55:12,568 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:12,568 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 12d8d2c95ad908a17b68eef8e95649aa/B is initiating minor compaction (all files) 2023-03-23 19:55:12,568 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12d8d2c95ad908a17b68eef8e95649aa/B in TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,568 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp, totalSize=46.4 K 2023-03-23 19:55:12,569 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 22e6de6cc05449418578fd6dfa01e0b2, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679601307661 2023-03-23 19:55:12,569 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e76bfab757ad416d94a824d853b6d84d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679601307662 2023-03-23 19:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:12,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:12,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:12,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:12,569 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting eaff0b0e86174350bc79295cdbf6a2b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601307694 2023-03-23 19:55:12,569 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b2ab7d9e7cab4fcca6cdccce115bfb75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601307694 2023-03-23 19:55:12,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:12,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:12,570 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9884a4b784204d1195b74a425e53f4f4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601309840 2023-03-23 19:55:12,570 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c29b5f07634942eba3650f9dab3ccaf6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601309840 2023-03-23 19:55:12,570 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a22379685cbc41228c554482bc0ad2cf, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679601310972 2023-03-23 19:55:12,570 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ce50a23b25aa446f8e9c116d2088e9fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679601310972 2023-03-23 19:55:12,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,572 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:12,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:12,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:12,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:12,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:12,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,577 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:12,577 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:12,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:12,587 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 12d8d2c95ad908a17b68eef8e95649aa#B#compaction#328 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:12,588 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12d8d2c95ad908a17b68eef8e95649aa#A#compaction#327 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:12,678 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:12,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:12,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:12,681 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:12,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:12,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:12,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:12,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:12,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,684 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:12,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:12,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:12,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:12,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:12,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:12,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:12,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:12,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:12,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:12,685 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:12,685 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:12,686 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:12,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:12,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:12,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601372785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601372792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601372793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601372794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601372794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601372897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601372904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601372904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601372904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:12,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601372904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,002 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/e5a44e448be54453b8e7963e29d8fe8c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e5a44e448be54453b8e7963e29d8fe8c 2023-03-23 19:55:13,009 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 12d8d2c95ad908a17b68eef8e95649aa/B of 12d8d2c95ad908a17b68eef8e95649aa into e5a44e448be54453b8e7963e29d8fe8c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:13,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:13,009 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., storeName=12d8d2c95ad908a17b68eef8e95649aa/B, priority=12, startTime=1679601312566; duration=0sec 2023-03-23 19:55:13,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:13,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:13,010 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:13,010 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 12d8d2c95ad908a17b68eef8e95649aa/C is initiating minor compaction (all files) 2023-03-23 19:55:13,010 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12d8d2c95ad908a17b68eef8e95649aa/C in TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:13,010 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp, totalSize=46.4 K 2023-03-23 19:55:13,011 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2d5fade8c08f4a8aafe7076db92b649a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1679601307662 2023-03-23 19:55:13,011 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 774497d3915d4d2086e4c31052bb287e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679601307694 2023-03-23 19:55:13,011 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting fc7ae564b92242a7a31b9224c143c98d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601309840 2023-03-23 19:55:13,012 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 58a95273aad440ba96f62a996004a87e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679601310972 2023-03-23 19:55:13,025 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 12d8d2c95ad908a17b68eef8e95649aa#C#compaction#330 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:13,036 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/428aa9a01e1c451aae8ff1883e4a849e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/428aa9a01e1c451aae8ff1883e4a849e 2023-03-23 19:55:13,043 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 12d8d2c95ad908a17b68eef8e95649aa/C of 12d8d2c95ad908a17b68eef8e95649aa into 428aa9a01e1c451aae8ff1883e4a849e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:13,043 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:13,043 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., storeName=12d8d2c95ad908a17b68eef8e95649aa/C, priority=12, startTime=1679601312567; duration=0sec 2023-03-23 19:55:13,043 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:13,099 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323ed766bd0376d48548de3b8b3cb8826ad to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ed766bd0376d48548de3b8b3cb8826ad 2023-03-23 19:55:13,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601373102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601373109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601373110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601373110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601373110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601373407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601373416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601373416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,420 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/09d3e33aa17a499fa50c43c563db8d50 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/09d3e33aa17a499fa50c43c563db8d50 2023-03-23 19:55:13,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601373417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601373418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,424 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 12d8d2c95ad908a17b68eef8e95649aa/A of 12d8d2c95ad908a17b68eef8e95649aa into 09d3e33aa17a499fa50c43c563db8d50(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:13,424 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:13,424 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., storeName=12d8d2c95ad908a17b68eef8e95649aa/A, priority=12, startTime=1679601312566; duration=0sec 2023-03-23 19:55:13,425 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:13,504 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/607b31aaae3d42cc964c1f7d25f47ca9 2023-03-23 19:55:13,915 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d 2023-03-23 19:55:13,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601373916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601373921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601373921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601373922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:13,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:13,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601373923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,326 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/b92adc22616046b98be19d08eea8526c 2023-03-23 19:55:14,330 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/607b31aaae3d42cc964c1f7d25f47ca9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/607b31aaae3d42cc964c1f7d25f47ca9 2023-03-23 19:55:14,334 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/607b31aaae3d42cc964c1f7d25f47ca9, entries=150, sequenceid=83, filesize=25.1 K 2023-03-23 19:55:14,334 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d 2023-03-23 19:55:14,338 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d, entries=150, sequenceid=83, filesize=11.6 K 2023-03-23 19:55:14,339 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/b92adc22616046b98be19d08eea8526c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/b92adc22616046b98be19d08eea8526c 2023-03-23 19:55:14,342 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/b92adc22616046b98be19d08eea8526c, entries=150, sequenceid=83, filesize=11.6 K 2023-03-23 19:55:14,342 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 12d8d2c95ad908a17b68eef8e95649aa in 1657ms, sequenceid=83, compaction requested=false 2023-03-23 19:55:14,342 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:14,342 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:14,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:14,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:14,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:14,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:14,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:14,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:14,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60dc53e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:14,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:14,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,348 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:14,349 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:14,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,349 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:14,350 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:14,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,350 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:14,350 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:14,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:14,352 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:14,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:14,353 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:14,354 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,354 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:14,354 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:14,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:14,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,356 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:14,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:14,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:14,360 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:14,360 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:14,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:14,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:14,360 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:14,360 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:14,366 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:14,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:14,460 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:14,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:14,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:14,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:14,464 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:14,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:14,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:14,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:14,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:14,466 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:14,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:14,466 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:14,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:14,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:14,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:14,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:14,468 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:14,468 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:14,468 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:14,469 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:14,888 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323e833ef7ebe594f9fa8d708859933e93c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e833ef7ebe594f9fa8d708859933e93c 2023-03-23 19:55:14,908 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/940b64e9b5d84b5c9fb1d5301e739186 2023-03-23 19:55:14,925 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/4bb166c14a3b42dbbe26517091376d07 2023-03-23 19:55:14,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:14,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601374941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601374941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601374942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601374945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:14,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601374946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601375049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601375049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601375049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601375049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601375052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601375255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601375255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601375255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601375255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601375256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,341 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/628c71dc17c24550aa8a57220843c340 2023-03-23 19:55:15,345 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/940b64e9b5d84b5c9fb1d5301e739186 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/940b64e9b5d84b5c9fb1d5301e739186 2023-03-23 19:55:15,349 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/940b64e9b5d84b5c9fb1d5301e739186, entries=150, sequenceid=105, filesize=25.1 K 2023-03-23 19:55:15,349 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/4bb166c14a3b42dbbe26517091376d07 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4bb166c14a3b42dbbe26517091376d07 2023-03-23 19:55:15,353 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4bb166c14a3b42dbbe26517091376d07, entries=150, sequenceid=105, filesize=11.6 K 2023-03-23 19:55:15,353 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/628c71dc17c24550aa8a57220843c340 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/628c71dc17c24550aa8a57220843c340 2023-03-23 19:55:15,356 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/628c71dc17c24550aa8a57220843c340, entries=150, sequenceid=105, filesize=11.6 K 2023-03-23 19:55:15,357 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 12d8d2c95ad908a17b68eef8e95649aa in 889ms, sequenceid=105, compaction requested=true 2023-03-23 19:55:15,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:15,357 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:15,357 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:15,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:15,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:15,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:15,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43760e4d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:15,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:15,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,362 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:15,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:15,362 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,363 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:15,367 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:15,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:15,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:15,370 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:15,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:15,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:15,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:15,370 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:15,371 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,371 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:15,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,371 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:15,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:15,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:15,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:15,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:15,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,376 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:15,376 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:15,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:15,476 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:15,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:15,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:15,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:15,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:15,480 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:15,480 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:15,480 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:15,480 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:15,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:15,484 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,484 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:15,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:15,484 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,484 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:15,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:15,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:15,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:15,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:15,485 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:15,488 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:15,488 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:15,488 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:15,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:15,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:15,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:15,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601375581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601375581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601375582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601375583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601375586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601375690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601375690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601375690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601375690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601375693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601375896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601375896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601375896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601375897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601375897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:15,904 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323c2bf81cab37f4d6f970c41fbba0f02dd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323c2bf81cab37f4d6f970c41fbba0f02dd 2023-03-23 19:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601376201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601376201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601376201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601376201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601376203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,309 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/46f2a92ad5794c9dafe0344b226b4be9 2023-03-23 19:55:16,318 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/6ad3861cf58b4bb0b51747dabc562981 2023-03-23 19:55:16,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601376705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601376705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601376706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601376707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:16,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601376713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,728 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/c91c186ee3b2457887191549450f3efe 2023-03-23 19:55:16,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/46f2a92ad5794c9dafe0344b226b4be9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/46f2a92ad5794c9dafe0344b226b4be9 2023-03-23 19:55:16,734 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/46f2a92ad5794c9dafe0344b226b4be9, entries=150, sequenceid=118, filesize=25.1 K 2023-03-23 19:55:16,735 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/6ad3861cf58b4bb0b51747dabc562981 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/6ad3861cf58b4bb0b51747dabc562981 2023-03-23 19:55:16,738 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/6ad3861cf58b4bb0b51747dabc562981, entries=150, sequenceid=118, filesize=11.6 K 2023-03-23 19:55:16,738 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/c91c186ee3b2457887191549450f3efe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c91c186ee3b2457887191549450f3efe 2023-03-23 19:55:16,741 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c91c186ee3b2457887191549450f3efe, entries=150, sequenceid=118, filesize=11.6 K 2023-03-23 19:55:16,742 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 12d8d2c95ad908a17b68eef8e95649aa in 1254ms, sequenceid=118, compaction requested=true 2023-03-23 19:55:16,742 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:16,742 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:16,742 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,745 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:16,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a815e2d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:16,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:16,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,748 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,748 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:16,748 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:16,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,748 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:16,748 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:16,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,750 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:16,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:16,752 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:16,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:16,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:16,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:16,752 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:16,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:16,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:16,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,756 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:16,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:16,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:16,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:16,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:16,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,759 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:16,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:16,760 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:16,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,860 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:16,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:16,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:16,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:16,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:16,864 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:16,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:16,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:16,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:16,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:16,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,866 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:16,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:16,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:16,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:16,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:16,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:16,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:16,867 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:16,867 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:16,867 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:16,867 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:16,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:16,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:16,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:16,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:16,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:17,282 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323ad1ef8e38ae14210b44c804817bf36b9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ad1ef8e38ae14210b44c804817bf36b9 2023-03-23 19:55:17,687 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/ff6d8551c69c410e98d05b19a22e2010 2023-03-23 19:55:17,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:17,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:17,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601377725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601377725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601377725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601377726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601377729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601377833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601377833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601377833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:17,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:17,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601377836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601378039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601378039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601378040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601378043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,097 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/d5b6c8c16af64a1dbc2057a23e61a30e 2023-03-23 19:55:18,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601378347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601378347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601378347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601378349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,507 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/198c06cb95874fd7a41afc019b85c526 2023-03-23 19:55:18,511 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/ff6d8551c69c410e98d05b19a22e2010 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ff6d8551c69c410e98d05b19a22e2010 2023-03-23 19:55:18,516 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ff6d8551c69c410e98d05b19a22e2010, entries=150, sequenceid=137, filesize=25.3 K 2023-03-23 19:55:18,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/d5b6c8c16af64a1dbc2057a23e61a30e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/d5b6c8c16af64a1dbc2057a23e61a30e 2023-03-23 19:55:18,520 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/d5b6c8c16af64a1dbc2057a23e61a30e, entries=150, sequenceid=137, filesize=11.8 K 2023-03-23 19:55:18,521 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/198c06cb95874fd7a41afc019b85c526 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/198c06cb95874fd7a41afc019b85c526 2023-03-23 19:55:18,524 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/198c06cb95874fd7a41afc019b85c526, entries=150, sequenceid=137, filesize=11.8 K 2023-03-23 19:55:18,525 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 12d8d2c95ad908a17b68eef8e95649aa in 1658ms, sequenceid=137, compaction requested=true 2023-03-23 19:55:18,525 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:18,525 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4fbb71[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:18,529 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:18,529 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,530 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:18,530 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:18,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:18,531 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,531 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:18,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:18,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:18,532 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,532 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:18,536 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:18,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:18,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:18,536 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:18,536 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:18,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:18,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,539 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:18,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:18,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:18,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,541 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:18,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:18,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:18,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:18,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:18,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:18,544 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:18,544 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:18,644 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:18,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:18,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:18,647 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:18,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:18,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:18,647 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:18,648 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,648 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:18,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:18,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:18,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:18,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:18,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:18,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:18,650 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:18,651 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:18,651 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:18,651 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:18,651 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:18,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:18,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:18,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601378883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601378888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601378890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601378891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:18,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601378992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601378996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601378998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601378998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,068 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032387c7588e14c546b0a78cc512ee04d12a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032387c7588e14c546b0a78cc512ee04d12a 2023-03-23 19:55:19,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601379200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601379202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601379204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601379204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,473 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/cf10079ef46f42cdae546432d36740fe 2023-03-23 19:55:19,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601379506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601379509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601379510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601379512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:19,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601379751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:19,758 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:55:19,883 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/42751420d3004e35bf3cb429a085cce4 2023-03-23 19:55:20,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:20,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601380012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601380017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601380018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:20,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601380019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,292 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/c29b6e5c0f9b40fbaf9328f63302257c 2023-03-23 19:55:20,305 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/cf10079ef46f42cdae546432d36740fe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/cf10079ef46f42cdae546432d36740fe 2023-03-23 19:55:20,309 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/cf10079ef46f42cdae546432d36740fe, entries=150, sequenceid=150, filesize=25.3 K 2023-03-23 19:55:20,310 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/42751420d3004e35bf3cb429a085cce4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/42751420d3004e35bf3cb429a085cce4 2023-03-23 19:55:20,314 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/42751420d3004e35bf3cb429a085cce4, entries=150, sequenceid=150, filesize=11.8 K 2023-03-23 19:55:20,314 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/c29b6e5c0f9b40fbaf9328f63302257c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c29b6e5c0f9b40fbaf9328f63302257c 2023-03-23 19:55:20,318 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c29b6e5c0f9b40fbaf9328f63302257c, entries=150, sequenceid=150, filesize=11.8 K 2023-03-23 19:55:20,319 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 12d8d2c95ad908a17b68eef8e95649aa in 1668ms, sequenceid=150, compaction requested=true 2023-03-23 19:55:20,319 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:20,319 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:20,321 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,321 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3603b1b3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:20,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:20,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,327 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:20,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:20,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:20,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:20,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:20,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:20,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:20,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:20,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:20,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:20,343 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:20,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,344 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:20,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:20,344 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:20,444 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:20,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:20,447 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:20,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:20,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:20,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:20,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:20,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:20,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:20,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:20,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:20,451 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:20,451 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:20,451 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:20,451 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:20,452 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:20,463 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303237324e83f389d4d55b41ef00074ef0694 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303237324e83f389d4d55b41ef00074ef0694 2023-03-23 19:55:20,874 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/b91544bb83404b73899b2e21318e4e21 2023-03-23 19:55:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:21,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601381042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601381043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601381044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601381046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601381153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601381153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601381153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601381153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,284 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/c100fc63872e4b178afa789d78f98652 2023-03-23 19:55:21,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601381361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601381361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601381361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601381362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601381670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601381670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601381671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:21,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601381672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,693 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/9455ce90accd49dd8be9ca61663b18e8 2023-03-23 19:55:21,697 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/b91544bb83404b73899b2e21318e4e21 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b91544bb83404b73899b2e21318e4e21 2023-03-23 19:55:21,700 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b91544bb83404b73899b2e21318e4e21, entries=150, sequenceid=169, filesize=25.3 K 2023-03-23 19:55:21,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/c100fc63872e4b178afa789d78f98652 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c100fc63872e4b178afa789d78f98652 2023-03-23 19:55:21,704 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c100fc63872e4b178afa789d78f98652, entries=150, sequenceid=169, filesize=11.8 K 2023-03-23 19:55:21,705 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/9455ce90accd49dd8be9ca61663b18e8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/9455ce90accd49dd8be9ca61663b18e8 2023-03-23 19:55:21,709 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/9455ce90accd49dd8be9ca61663b18e8, entries=150, sequenceid=169, filesize=11.8 K 2023-03-23 19:55:21,710 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 12d8d2c95ad908a17b68eef8e95649aa in 1259ms, sequenceid=169, compaction requested=true 2023-03-23 19:55:21,710 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:21,710 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:21,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f4c558[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:21,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:21,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,715 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,716 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:21,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:21,720 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:21,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:21,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:21,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:21,720 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:21,722 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,722 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,722 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:21,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:21,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:21,726 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:21,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:21,726 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:21,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:21,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:21,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:21,826 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:21,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:21,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:21,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:21,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:21,830 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:21,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:21,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:21,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:21,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:21,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:21,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:21,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:21,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:21,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:21,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:21,834 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:21,834 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:21,834 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:21,834 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:21,834 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:21,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:21,866 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323e3560f8ffc7a46a8b042f06fdc48b998 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e3560f8ffc7a46a8b042f06fdc48b998 2023-03-23 19:55:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:22,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:22,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601382213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601382213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601382221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601382222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,278 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/b80ef0b911304e2180a720f6a24a6c35 2023-03-23 19:55:22,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601382323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601382323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601382330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601382330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57068 deadline: 1679601382528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57102 deadline: 1679601382529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57084 deadline: 1679601382537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601382539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,687 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/cb37cd83ed6041a9b6f731494eea25d1 2023-03-23 19:55:22,697 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/7ab2407f46e7488892edef86f10f266e 2023-03-23 19:55:22,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/b80ef0b911304e2180a720f6a24a6c35 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b80ef0b911304e2180a720f6a24a6c35 2023-03-23 19:55:22,704 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b80ef0b911304e2180a720f6a24a6c35, entries=150, sequenceid=182, filesize=25.3 K 2023-03-23 19:55:22,705 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/cb37cd83ed6041a9b6f731494eea25d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/cb37cd83ed6041a9b6f731494eea25d1 2023-03-23 19:55:22,709 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/cb37cd83ed6041a9b6f731494eea25d1, entries=150, sequenceid=182, filesize=11.8 K 2023-03-23 19:55:22,709 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/7ab2407f46e7488892edef86f10f266e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7ab2407f46e7488892edef86f10f266e 2023-03-23 19:55:22,712 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7ab2407f46e7488892edef86f10f266e, entries=150, sequenceid=182, filesize=11.8 K 2023-03-23 19:55:22,713 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 12d8d2c95ad908a17b68eef8e95649aa in 879ms, sequenceid=182, compaction requested=true 2023-03-23 19:55:22,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:22,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:22,715 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75348bfd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:22,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:22,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,719 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:22,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:22,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:22,723 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:22,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:22,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:22,723 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:22,723 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:22,725 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,725 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,725 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:22,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:22,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:22,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:22,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:22,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:22,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,730 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:22,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:22,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:22,731 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:22,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:22,831 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:22,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:22,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:22,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:22,834 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:22,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:22,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:22,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:22,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:22,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:22,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:22,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:22,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:22,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:22,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:22,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:22,837 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:22,838 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:22,838 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:22,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:22,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601382857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601382867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601382868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601382869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601382974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601382984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601382984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:22,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:22,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601382984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57112 deadline: 1679601383183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601383192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601383192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601383195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,249 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323abcb776e873647ceab69f3c0e2285fd8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323abcb776e873647ceab69f3c0e2285fd8 2023-03-23 19:55:23,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601383493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601383503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601383504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601383504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,654 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/44e9d9808e964bf39b04c1b2bd494fb8 2023-03-23 19:55:23,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:23,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57088 deadline: 1679601383776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:23,782 DEBUG [Thread-2310] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:55:24,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601384002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601384015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:24,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601384016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:24,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601384017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,065 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/23829f7c6daa4b3d893a3ae0d60b2dcf 2023-03-23 19:55:24,474 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/ca76a0971b7a477d90b5767fc365afd1 2023-03-23 19:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/44e9d9808e964bf39b04c1b2bd494fb8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/44e9d9808e964bf39b04c1b2bd494fb8 2023-03-23 19:55:24,481 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/44e9d9808e964bf39b04c1b2bd494fb8, entries=150, sequenceid=202, filesize=25.3 K 2023-03-23 19:55:24,482 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/23829f7c6daa4b3d893a3ae0d60b2dcf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/23829f7c6daa4b3d893a3ae0d60b2dcf 2023-03-23 19:55:24,485 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/23829f7c6daa4b3d893a3ae0d60b2dcf, entries=150, sequenceid=202, filesize=11.8 K 2023-03-23 19:55:24,486 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/ca76a0971b7a477d90b5767fc365afd1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/ca76a0971b7a477d90b5767fc365afd1 2023-03-23 19:55:24,490 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/ca76a0971b7a477d90b5767fc365afd1, entries=150, sequenceid=202, filesize=11.8 K 2023-03-23 19:55:24,490 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 12d8d2c95ad908a17b68eef8e95649aa in 1652ms, sequenceid=202, compaction requested=true 2023-03-23 19:55:24,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:24,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:24,492 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:24,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:24,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56033219[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:24,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:24,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:24,495 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,495 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:24,497 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,497 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:24,497 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:24,497 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:24,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:24,499 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:24,499 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:24,500 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:24,500 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:24,500 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:24,501 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,501 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,501 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:24,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:24,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:24,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,507 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,507 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:24,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:24,508 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,508 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:24,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:24,508 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:24,508 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:24,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:24,608 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:24,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:24,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:24,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:24,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:24,611 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:24,614 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,614 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:24,615 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:24,615 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:24,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:24,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:24,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:24,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:24,616 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:24,616 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:24,616 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:24,629 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323ec43bf813ffb444daf15ad7559c7320a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ec43bf813ffb444daf15ad7559c7320a 2023-03-23 19:55:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:25,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:25,036 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/5c8b152fa54c4d4ea676136699f8353e 2023-03-23 19:55:25,045 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/31fdb37f279645c2825f4924ff16986b 2023-03-23 19:55:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601385074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601385075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601385079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601385080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601385186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601385186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601385191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601385191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601385390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601385390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601385401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601385401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,453 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/1981889a7f9644f9b19fb97859409f57 2023-03-23 19:55:25,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/5c8b152fa54c4d4ea676136699f8353e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/5c8b152fa54c4d4ea676136699f8353e 2023-03-23 19:55:25,460 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/5c8b152fa54c4d4ea676136699f8353e, entries=150, sequenceid=214, filesize=25.3 K 2023-03-23 19:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/31fdb37f279645c2825f4924ff16986b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/31fdb37f279645c2825f4924ff16986b 2023-03-23 19:55:25,464 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/31fdb37f279645c2825f4924ff16986b, entries=150, sequenceid=214, filesize=11.8 K 2023-03-23 19:55:25,465 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/1981889a7f9644f9b19fb97859409f57 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/1981889a7f9644f9b19fb97859409f57 2023-03-23 19:55:25,468 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/1981889a7f9644f9b19fb97859409f57, entries=150, sequenceid=214, filesize=11.8 K 2023-03-23 19:55:25,468 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 12d8d2c95ad908a17b68eef8e95649aa in 852ms, sequenceid=214, compaction requested=true 2023-03-23 19:55:25,468 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:25,468 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:25,469 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:25,469 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:25,469 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,469 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:25,469 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:25,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:25,471 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d269f43[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:25,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:25,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,473 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:25,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,473 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:25,474 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:25,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:25,477 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:25,477 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:25,477 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:25,477 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:25,477 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:25,479 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,479 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,479 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:25,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:25,480 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:25,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:25,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:25,483 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:25,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:25,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:25,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:25,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:25,483 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:25,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:25,483 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:25,484 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:25,484 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:25,484 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:25,484 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,584 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:25,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:25,587 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:25,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:25,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:25,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:25,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:25,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:25,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:25,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:25,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:25,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:25,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:25,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:25,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:25,591 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:25,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:25,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:25,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601385727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601385728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601385731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601385740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601385840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601385842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601385848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:25,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601385848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,009 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032358db3a61041246529f9740e22c433c2c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358db3a61041246529f9740e22c433c2c 2023-03-23 19:55:26,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601386052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601386052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601386053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601386053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601386358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601386364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601386364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601386366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,414 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=234, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/ec555fbfaab84168a408aaf5e3a0a3ae 2023-03-23 19:55:26,824 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/205cfa0922804a8cb56154d8fa3dedaf 2023-03-23 19:55:26,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57112 deadline: 1679601386869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57068 deadline: 1679601386873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57102 deadline: 1679601386874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:26,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:26,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57084 deadline: 1679601386878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,234 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/7bfbbec2ff044ea3a0a6a8fdec886cc5 2023-03-23 19:55:27,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/ec555fbfaab84168a408aaf5e3a0a3ae as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ec555fbfaab84168a408aaf5e3a0a3ae 2023-03-23 19:55:27,241 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ec555fbfaab84168a408aaf5e3a0a3ae, entries=150, sequenceid=234, filesize=25.3 K 2023-03-23 19:55:27,242 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/205cfa0922804a8cb56154d8fa3dedaf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/205cfa0922804a8cb56154d8fa3dedaf 2023-03-23 19:55:27,245 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/205cfa0922804a8cb56154d8fa3dedaf, entries=150, sequenceid=234, filesize=11.8 K 2023-03-23 19:55:27,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/7bfbbec2ff044ea3a0a6a8fdec886cc5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7bfbbec2ff044ea3a0a6a8fdec886cc5 2023-03-23 19:55:27,250 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7bfbbec2ff044ea3a0a6a8fdec886cc5, entries=150, sequenceid=234, filesize=11.8 K 2023-03-23 19:55:27,250 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 12d8d2c95ad908a17b68eef8e95649aa in 1659ms, sequenceid=234, compaction requested=true 2023-03-23 19:55:27,251 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:27,251 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:27,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:27,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:27,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:27,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:27,253 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:27,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:27,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e1b5bba[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:27,256 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:27,256 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,258 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:27,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:27,259 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:27,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:27,262 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:27,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:27,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:27,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:27,262 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:27,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,266 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:27,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:27,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:27,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:27,268 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:27,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,268 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:27,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:27,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:27,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,269 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:27,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:27,369 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:27,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:27,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:27,373 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,373 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:27,375 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:27,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:27,376 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:27,376 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:27,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:27,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:27,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:27,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:27,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:27,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:27,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. started... 2023-03-23 19:55:27,377 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:55:27,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:27,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:27,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:27,378 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:27,378 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:27,378 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:27,638 DEBUG [Thread-2321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec116b to 127.0.0.1:60301 2023-03-23 19:55:27,638 DEBUG [Thread-2321] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,641 DEBUG [Thread-2319] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x159c5e0c to 127.0.0.1:60301 2023-03-23 19:55:27,642 DEBUG [Thread-2319] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,645 DEBUG [Thread-2313] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:60301 2023-03-23 19:55:27,645 DEBUG [Thread-2313] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,648 DEBUG [Thread-2317] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:60301 2023-03-23 19:55:27,648 DEBUG [Thread-2317] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,652 DEBUG [Thread-2315] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:60301 2023-03-23 19:55:27,652 DEBUG [Thread-2315] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,790 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323378c13ea6a3b494088a4b5b37aba2427 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323378c13ea6a3b494088a4b5b37aba2427 2023-03-23 19:55:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:27,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. as already flushing 2023-03-23 19:55:27,881 DEBUG [Thread-2306] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:60301 2023-03-23 19:55:27,881 DEBUG [Thread-2306] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,882 DEBUG [Thread-2304] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:60301 2023-03-23 19:55:27,882 DEBUG [Thread-2304] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,885 DEBUG [Thread-2302] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:60301 2023-03-23 19:55:27,886 DEBUG [Thread-2302] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:27,891 DEBUG [Thread-2308] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:60301 2023-03-23 19:55:27,891 DEBUG [Thread-2308] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:28,194 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/439da2ae1caf4a3eaf4fea68b20df6e0 2023-03-23 19:55:28,603 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/bb7161ff83434445a6da7ec771381dfe 2023-03-23 19:55:29,011 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/d66adcdba8274a8ba795f8aaeda39f1b 2023-03-23 19:55:29,015 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/439da2ae1caf4a3eaf4fea68b20df6e0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/439da2ae1caf4a3eaf4fea68b20df6e0 2023-03-23 19:55:29,017 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/439da2ae1caf4a3eaf4fea68b20df6e0, entries=150, sequenceid=246, filesize=25.3 K 2023-03-23 19:55:29,018 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/bb7161ff83434445a6da7ec771381dfe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/bb7161ff83434445a6da7ec771381dfe 2023-03-23 19:55:29,021 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/bb7161ff83434445a6da7ec771381dfe, entries=150, sequenceid=246, filesize=11.8 K 2023-03-23 19:55:29,021 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/d66adcdba8274a8ba795f8aaeda39f1b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/d66adcdba8274a8ba795f8aaeda39f1b 2023-03-23 19:55:29,024 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/d66adcdba8274a8ba795f8aaeda39f1b, entries=150, sequenceid=246, filesize=11.8 K 2023-03-23 19:55:29,025 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 12d8d2c95ad908a17b68eef8e95649aa in 1648ms, sequenceid=246, compaction requested=true 2023-03-23 19:55:29,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:29,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,026 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2150e7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:29,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:29,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,029 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:29,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,029 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:29,030 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:29,032 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:29,032 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:29,032 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:29,032 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:29,032 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:29,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:29,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:29,035 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:29,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:29,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:29,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:29,044 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:29,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:29,044 DEBUG [Thread-2312] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,044 DEBUG [Thread-2312] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:29,044 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:29,144 DEBUG [Thread-2312] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:33,843 DEBUG [Thread-2310] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:60301 2023-03-23 19:55:33,843 DEBUG [Thread-2310] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 43 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 1846 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 5538 rows 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 1840 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 5520 rows 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 1852 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 5556 rows 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 5604 rows 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 1842 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 5523 rows 2023-03-23 19:55:33,844 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:55:33,844 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:60301 2023-03-23 19:55:33,845 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:33,848 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:55:33,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:55:33,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:33,852 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601333852"}]},"ts":"1679601333852"} 2023-03-23 19:55:33,853 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:55:33,854 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:55:33,855 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, UNASSIGN}] 2023-03-23 19:55:33,856 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, UNASSIGN 2023-03-23 19:55:33,857 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:33,857 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601333856"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601333856"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601333856"}]},"ts":"1679601333856"} 2023-03-23 19:55:33,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:33,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:34,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 12d8d2c95ad908a17b68eef8e95649aa, disabling compactions & flushes 2023-03-23 19:55:34,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. after waiting 0 ms 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:34,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 12d8d2c95ad908a17b68eef8e95649aa 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=A 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=B 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:34,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12d8d2c95ad908a17b68eef8e95649aa, store=C 2023-03-23 19:55:34,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:34,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:34,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303239f2ccbd88a854bea8acc5163f87350f1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f2ccbd88a854bea8acc5163f87350f1 2023-03-23 19:55:34,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:34,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/a1a9970fb7924b49a0c8c3c1866cc1e4 2023-03-23 19:55:34,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:35,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/4124543e5f52493a81699bc58c1eeee5 2023-03-23 19:55:35,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/70f7d6766bb94e06b26525f01015c0b3 2023-03-23 19:55:35,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/A/a1a9970fb7924b49a0c8c3c1866cc1e4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a1a9970fb7924b49a0c8c3c1866cc1e4 2023-03-23 19:55:35,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a1a9970fb7924b49a0c8c3c1866cc1e4, entries=150, sequenceid=254, filesize=25.3 K 2023-03-23 19:55:35,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/B/4124543e5f52493a81699bc58c1eeee5 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4124543e5f52493a81699bc58c1eeee5 2023-03-23 19:55:35,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4124543e5f52493a81699bc58c1eeee5, entries=150, sequenceid=254, filesize=11.8 K 2023-03-23 19:55:35,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/.tmp/C/70f7d6766bb94e06b26525f01015c0b3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/70f7d6766bb94e06b26525f01015c0b3 2023-03-23 19:55:35,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/70f7d6766bb94e06b26525f01015c0b3, entries=150, sequenceid=254, filesize=11.8 K 2023-03-23 19:55:35,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 12d8d2c95ad908a17b68eef8e95649aa in 1646ms, sequenceid=254, compaction requested=true 2023-03-23 19:55:35,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf] to archive 2023-03-23 19:55:35,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:35,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/22e6de6cc05449418578fd6dfa01e0b2 2023-03-23 19:55:35,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/eaff0b0e86174350bc79295cdbf6a2b6 2023-03-23 19:55:35,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/9884a4b784204d1195b74a425e53f4f4 2023-03-23 19:55:35,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a22379685cbc41228c554482bc0ad2cf 2023-03-23 19:55:35,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa] to archive 2023-03-23 19:55:35,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:35,684 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e76bfab757ad416d94a824d853b6d84d 2023-03-23 19:55:35,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/b2ab7d9e7cab4fcca6cdccce115bfb75 2023-03-23 19:55:35,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c29b5f07634942eba3650f9dab3ccaf6 2023-03-23 19:55:35,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/ce50a23b25aa446f8e9c116d2088e9fa 2023-03-23 19:55:35,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e] to archive 2023-03-23 19:55:35,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:55:35,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/2d5fade8c08f4a8aafe7076db92b649a 2023-03-23 19:55:35,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/774497d3915d4d2086e4c31052bb287e 2023-03-23 19:55:35,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/fc7ae564b92242a7a31b9224c143c98d 2023-03-23 19:55:35,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/58a95273aad440ba96f62a996004a87e 2023-03-23 19:55:35,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits/257.seqid, newMaxSeqId=257, maxSeqId=4 2023-03-23 19:55:35,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa. 2023-03-23 19:55:35,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 12d8d2c95ad908a17b68eef8e95649aa: 2023-03-23 19:55:35,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:35,698 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=12d8d2c95ad908a17b68eef8e95649aa, regionState=CLOSED 2023-03-23 19:55:35,698 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601335698"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601335698"}]},"ts":"1679601335698"} 2023-03-23 19:55:35,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-23 19:55:35,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 12d8d2c95ad908a17b68eef8e95649aa, server=jenkins-hbase4.apache.org,34277,1679601218015 in 1.8420 sec 2023-03-23 19:55:35,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-23 19:55:35,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12d8d2c95ad908a17b68eef8e95649aa, UNASSIGN in 1.8450 sec 2023-03-23 19:55:35,702 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601335702"}]},"ts":"1679601335702"} 2023-03-23 19:55:35,702 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:55:35,705 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:55:35,706 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8570 sec 2023-03-23 19:55:35,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-23 19:55:35,956 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-23 19:55:35,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:55:35,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:35,959 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-23 19:55:35,959 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:35,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:35,966 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits] 2023-03-23 19:55:35,968 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/09d3e33aa17a499fa50c43c563db8d50 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/09d3e33aa17a499fa50c43c563db8d50 2023-03-23 19:55:35,969 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/439da2ae1caf4a3eaf4fea68b20df6e0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/439da2ae1caf4a3eaf4fea68b20df6e0 2023-03-23 19:55:35,970 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/44e9d9808e964bf39b04c1b2bd494fb8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/44e9d9808e964bf39b04c1b2bd494fb8 2023-03-23 19:55:35,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/46f2a92ad5794c9dafe0344b226b4be9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/46f2a92ad5794c9dafe0344b226b4be9 2023-03-23 19:55:35,972 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/5c8b152fa54c4d4ea676136699f8353e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/5c8b152fa54c4d4ea676136699f8353e 2023-03-23 19:55:35,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/607b31aaae3d42cc964c1f7d25f47ca9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/607b31aaae3d42cc964c1f7d25f47ca9 2023-03-23 19:55:35,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/940b64e9b5d84b5c9fb1d5301e739186 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/940b64e9b5d84b5c9fb1d5301e739186 2023-03-23 19:55:35,974 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a1a9970fb7924b49a0c8c3c1866cc1e4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/a1a9970fb7924b49a0c8c3c1866cc1e4 2023-03-23 19:55:35,975 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b80ef0b911304e2180a720f6a24a6c35 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b80ef0b911304e2180a720f6a24a6c35 2023-03-23 19:55:35,976 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b91544bb83404b73899b2e21318e4e21 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/b91544bb83404b73899b2e21318e4e21 2023-03-23 19:55:35,977 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/cf10079ef46f42cdae546432d36740fe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/cf10079ef46f42cdae546432d36740fe 2023-03-23 19:55:35,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ec555fbfaab84168a408aaf5e3a0a3ae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ec555fbfaab84168a408aaf5e3a0a3ae 2023-03-23 19:55:35,979 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ff6d8551c69c410e98d05b19a22e2010 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/A/ff6d8551c69c410e98d05b19a22e2010 2023-03-23 19:55:35,982 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/205cfa0922804a8cb56154d8fa3dedaf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/205cfa0922804a8cb56154d8fa3dedaf 2023-03-23 19:55:35,983 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/23829f7c6daa4b3d893a3ae0d60b2dcf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/23829f7c6daa4b3d893a3ae0d60b2dcf 2023-03-23 19:55:35,984 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/31fdb37f279645c2825f4924ff16986b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/31fdb37f279645c2825f4924ff16986b 2023-03-23 19:55:35,985 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4124543e5f52493a81699bc58c1eeee5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4124543e5f52493a81699bc58c1eeee5 2023-03-23 19:55:35,986 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/42751420d3004e35bf3cb429a085cce4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/42751420d3004e35bf3cb429a085cce4 2023-03-23 19:55:35,987 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4bb166c14a3b42dbbe26517091376d07 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/4bb166c14a3b42dbbe26517091376d07 2023-03-23 19:55:35,988 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/6ad3861cf58b4bb0b51747dabc562981 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/6ad3861cf58b4bb0b51747dabc562981 2023-03-23 19:55:35,989 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/83133a7a0ccf4d8c9ec23aa9b46d8b2d 2023-03-23 19:55:35,990 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/bb7161ff83434445a6da7ec771381dfe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/bb7161ff83434445a6da7ec771381dfe 2023-03-23 19:55:35,991 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c100fc63872e4b178afa789d78f98652 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/c100fc63872e4b178afa789d78f98652 2023-03-23 19:55:35,991 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/cb37cd83ed6041a9b6f731494eea25d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/cb37cd83ed6041a9b6f731494eea25d1 2023-03-23 19:55:35,992 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/d5b6c8c16af64a1dbc2057a23e61a30e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/d5b6c8c16af64a1dbc2057a23e61a30e 2023-03-23 19:55:35,993 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e5a44e448be54453b8e7963e29d8fe8c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/B/e5a44e448be54453b8e7963e29d8fe8c 2023-03-23 19:55:35,995 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/1981889a7f9644f9b19fb97859409f57 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/1981889a7f9644f9b19fb97859409f57 2023-03-23 19:55:35,996 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/198c06cb95874fd7a41afc019b85c526 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/198c06cb95874fd7a41afc019b85c526 2023-03-23 19:55:35,997 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/428aa9a01e1c451aae8ff1883e4a849e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/428aa9a01e1c451aae8ff1883e4a849e 2023-03-23 19:55:35,998 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/628c71dc17c24550aa8a57220843c340 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/628c71dc17c24550aa8a57220843c340 2023-03-23 19:55:35,999 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/70f7d6766bb94e06b26525f01015c0b3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/70f7d6766bb94e06b26525f01015c0b3 2023-03-23 19:55:35,999 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7ab2407f46e7488892edef86f10f266e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7ab2407f46e7488892edef86f10f266e 2023-03-23 19:55:36,000 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7bfbbec2ff044ea3a0a6a8fdec886cc5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/7bfbbec2ff044ea3a0a6a8fdec886cc5 2023-03-23 19:55:36,001 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/9455ce90accd49dd8be9ca61663b18e8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/9455ce90accd49dd8be9ca61663b18e8 2023-03-23 19:55:36,002 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/b92adc22616046b98be19d08eea8526c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/b92adc22616046b98be19d08eea8526c 2023-03-23 19:55:36,003 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c29b6e5c0f9b40fbaf9328f63302257c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c29b6e5c0f9b40fbaf9328f63302257c 2023-03-23 19:55:36,004 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c91c186ee3b2457887191549450f3efe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/c91c186ee3b2457887191549450f3efe 2023-03-23 19:55:36,005 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/ca76a0971b7a477d90b5767fc365afd1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/ca76a0971b7a477d90b5767fc365afd1 2023-03-23 19:55:36,006 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/d66adcdba8274a8ba795f8aaeda39f1b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/C/d66adcdba8274a8ba795f8aaeda39f1b 2023-03-23 19:55:36,008 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits/257.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa/recovered.edits/257.seqid 2023-03-23 19:55:36,009 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/12d8d2c95ad908a17b68eef8e95649aa 2023-03-23 19:55:36,009 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:55:36,009 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:55:36,010 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-23 19:55:36,012 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323378c13ea6a3b494088a4b5b37aba2427 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323378c13ea6a3b494088a4b5b37aba2427 2023-03-23 19:55:36,013 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358db3a61041246529f9740e22c433c2c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358db3a61041246529f9740e22c433c2c 2023-03-23 19:55:36,014 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303237324e83f389d4d55b41ef00074ef0694 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303237324e83f389d4d55b41ef00074ef0694 2023-03-23 19:55:36,015 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032386ab0864223d4fc0ac9f0e42f1b2e1f6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032386ab0864223d4fc0ac9f0e42f1b2e1f6 2023-03-23 19:55:36,016 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032387c7588e14c546b0a78cc512ee04d12a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032387c7588e14c546b0a78cc512ee04d12a 2023-03-23 19:55:36,017 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f2ccbd88a854bea8acc5163f87350f1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239f2ccbd88a854bea8acc5163f87350f1 2023-03-23 19:55:36,018 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323abcb776e873647ceab69f3c0e2285fd8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323abcb776e873647ceab69f3c0e2285fd8 2023-03-23 19:55:36,019 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ac4505883cc542b3be7e71a1c94475b0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ac4505883cc542b3be7e71a1c94475b0 2023-03-23 19:55:36,019 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ad1ef8e38ae14210b44c804817bf36b9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ad1ef8e38ae14210b44c804817bf36b9 2023-03-23 19:55:36,020 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323c2bf81cab37f4d6f970c41fbba0f02dd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323c2bf81cab37f4d6f970c41fbba0f02dd 2023-03-23 19:55:36,021 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323caa8eaabd1c943c8844bc7ced1e611d7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323caa8eaabd1c943c8844bc7ced1e611d7 2023-03-23 19:55:36,022 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d8f046e34c1640e7be93ecf341165953 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d8f046e34c1640e7be93ecf341165953 2023-03-23 19:55:36,023 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e3560f8ffc7a46a8b042f06fdc48b998 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e3560f8ffc7a46a8b042f06fdc48b998 2023-03-23 19:55:36,024 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e833ef7ebe594f9fa8d708859933e93c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323e833ef7ebe594f9fa8d708859933e93c 2023-03-23 19:55:36,025 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ec43bf813ffb444daf15ad7559c7320a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ec43bf813ffb444daf15ad7559c7320a 2023-03-23 19:55:36,026 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ed766bd0376d48548de3b8b3cb8826ad to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323ed766bd0376d48548de3b8b3cb8826ad 2023-03-23 19:55:36,027 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:55:36,029 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:36,030 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:55:36,031 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:55:36,032 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:36,032 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:55:36,032 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601336032"}]},"ts":"9223372036854775807"} 2023-03-23 19:55:36,033 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:55:36,033 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 12d8d2c95ad908a17b68eef8e95649aa, NAME => 'TestAcidGuarantees,,1679601304548.12d8d2c95ad908a17b68eef8e95649aa.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:55:36,033 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:55:36,034 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601336034"}]},"ts":"9223372036854775807"} 2023-03-23 19:55:36,036 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:55:36,037 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:36,038 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 80 msec 2023-03-23 19:55:36,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-23 19:55:36,060 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-23 19:55:36,071 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224 (was 232), OpenFileDescriptor=502 (was 568), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=294 (was 368), ProcessCount=171 (was 171), AvailableMemoryMB=5370 (was 5394) 2023-03-23 19:55:36,082 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=224, OpenFileDescriptor=502, MaxFileDescriptor=60000, SystemLoadAverage=294, ProcessCount=171, AvailableMemoryMB=5370 2023-03-23 19:55:36,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:55:36,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:55:36,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:55:36,085 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:55:36,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-23 19:55:36,086 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:55:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:36,087 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:36,087 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 empty. 2023-03-23 19:55:36,088 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:36,088 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:55:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:36,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:36,494 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:55:36,496 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6b6146c7b551e398a832f6da27879933, NAME => 'TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:55:36,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6b6146c7b551e398a832f6da27879933, disabling compactions & flushes 2023-03-23 19:55:36,902 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. after waiting 0 ms 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:36,902 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:36,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:36,904 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:55:36,905 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601336905"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601336905"}]},"ts":"1679601336905"} 2023-03-23 19:55:36,906 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:55:36,906 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:55:36,907 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601336906"}]},"ts":"1679601336906"} 2023-03-23 19:55:36,907 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:55:36,910 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, ASSIGN}] 2023-03-23 19:55:36,912 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, ASSIGN 2023-03-23 19:55:36,912 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:55:37,063 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6b6146c7b551e398a832f6da27879933, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:37,063 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601337063"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601337063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601337063"}]},"ts":"1679601337063"} 2023-03-23 19:55:37,065 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:55:37,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:37,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:37,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b6146c7b551e398a832f6da27879933, NAME => 'TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:55:37,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:55:37,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,220 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,221 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A 2023-03-23 19:55:37,221 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A 2023-03-23 19:55:37,222 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:37,222 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b6146c7b551e398a832f6da27879933 columnFamilyName A 2023-03-23 19:55:37,222 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(310): Store=6b6146c7b551e398a832f6da27879933/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:37,222 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,223 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B 2023-03-23 19:55:37,223 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B 2023-03-23 19:55:37,223 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:37,224 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b6146c7b551e398a832f6da27879933 columnFamilyName B 2023-03-23 19:55:37,224 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(310): Store=6b6146c7b551e398a832f6da27879933/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:37,224 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,225 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C 2023-03-23 19:55:37,225 DEBUG [StoreOpener-6b6146c7b551e398a832f6da27879933-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C 2023-03-23 19:55:37,225 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:55:37,225 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b6146c7b551e398a832f6da27879933 columnFamilyName C 2023-03-23 19:55:37,225 INFO [StoreOpener-6b6146c7b551e398a832f6da27879933-1] regionserver.HStore(310): Store=6b6146c7b551e398a832f6da27879933/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:55:37,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:37,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:55:37,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:37,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:55:37,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6b6146c7b551e398a832f6da27879933; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73081038, jitterRate=0.08899232745170593}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:55:37,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:37,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., pid=49, masterSystemTime=1679601337216 2023-03-23 19:55:37,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:37,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:37,232 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6b6146c7b551e398a832f6da27879933, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:37,232 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601337232"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601337232"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601337232"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601337232"}]},"ts":"1679601337232"} 2023-03-23 19:55:37,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-23 19:55:37,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 in 168 msec 2023-03-23 19:55:37,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-23 19:55:37,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, ASSIGN in 324 msec 2023-03-23 19:55:37,235 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:55:37,235 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601337235"}]},"ts":"1679601337235"} 2023-03-23 19:55:37,236 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:55:37,239 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:55:37,240 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1550 sec 2023-03-23 19:55:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-23 19:55:38,190 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-23 19:55:38,191 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,196 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,197 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,199 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,200 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:55:38,201 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33960, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:55:38,202 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,206 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,206 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,211 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,211 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,216 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,217 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,220 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,221 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,224 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,224 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,228 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,228 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,232 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,232 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,236 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,236 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,240 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,240 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x4ac98310 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:55:38,244 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a9f3972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:55:38,249 DEBUG [hconnection-0x3c08e46c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,250 DEBUG [hconnection-0x3700a1aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,250 DEBUG [hconnection-0x3ee11dc6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,250 DEBUG [hconnection-0xbe10855-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,250 DEBUG [hconnection-0x2163ee82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,251 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35750, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,251 DEBUG [hconnection-0x41cd502a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,252 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,255 DEBUG [hconnection-0x7f44d244-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,257 DEBUG [hconnection-0x7d2cfade-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,258 DEBUG [hconnection-0x1f9367b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,258 DEBUG [hconnection-0x40265e47-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:55:38,259 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,262 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35756, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,262 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,262 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35766, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35830, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:55:38,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:38,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:55:38,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,277 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:38,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:38,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:38,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,277 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:38,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,286 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,286 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,287 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,288 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,288 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,288 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,290 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,290 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,291 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd30e0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601398290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601398291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601398292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,299 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,299 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,301 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601398296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601398297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,303 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,304 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,309 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,311 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,312 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,312 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,312 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601398398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601398398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601398398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601398402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601398403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,412 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:38,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:38,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,418 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,418 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,418 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,418 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,420 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,420 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,421 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e86b3b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,430 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,436 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,436 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,439 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,445 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:38,445 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,546 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:38,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:38,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,549 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,551 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,555 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dea3f37[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,560 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,562 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,562 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,566 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:38,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,569 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,569 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601398603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601398603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601398604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601398606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601398607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,669 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:38,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:38,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,672 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,674 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:38,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,676 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,676 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,676 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,676 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,678 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,679 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4809f005[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,681 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,685 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,686 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,686 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,689 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,692 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:38,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/8c9734a094834c9bb84877d137fd2148 2023-03-23 19:55:38,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,693 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,693 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,693 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/af73f36fd1f9406aa3acd2541fc1ec01 2023-03-23 19:55:38,793 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:38,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:38,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,796 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,797 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,798 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,798 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:38,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,800 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,800 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,802 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@219ab3d4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,805 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,805 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,805 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,811 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,813 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,814 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,817 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,819 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,819 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601398908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601398909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601398909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601398913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:38,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601398915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,920 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:38,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:38,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:38,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:38,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:38,923 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:38,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:38,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:38,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,933 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:38,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:38,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:38,936 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:38,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba1a56f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:38,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:38,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,939 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:38,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:38,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:38,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:38,943 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:38,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:38,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:38,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:38,944 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:38,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:38,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,951 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:38,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:38,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:38,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:38,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:38,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:38,956 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:38,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:38,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:38,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:38,956 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:38,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:38,957 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:38,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:38,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,057 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:39,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:39,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:39,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:39,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:39,060 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:39,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:39,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:39,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:39,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,063 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:39,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:39,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:39,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:39,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:39,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:39,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:39,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:39,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:39,064 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:39,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,066 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:39,067 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,067 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77ee34ab[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:39,068 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:39,068 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,070 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:39,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:39,070 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,070 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:39,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:39,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:39,074 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:39,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:39,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:39,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:39,074 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:39,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:39,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:39,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:39,076 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:39,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:39,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:39,080 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:39,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:39,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:39,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:39,081 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:39,081 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:39,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:39,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/689bedacd0334aff84cfb3f65a5ce14d 2023-03-23 19:55:39,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/8c9734a094834c9bb84877d137fd2148 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148 2023-03-23 19:55:39,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148, entries=150, sequenceid=15, filesize=11.6 K 2023-03-23 19:55:39,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/af73f36fd1f9406aa3acd2541fc1ec01 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01 2023-03-23 19:55:39,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01, entries=150, sequenceid=15, filesize=11.6 K 2023-03-23 19:55:39,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/689bedacd0334aff84cfb3f65a5ce14d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d 2023-03-23 19:55:39,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d, entries=150, sequenceid=15, filesize=11.6 K 2023-03-23 19:55:39,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6b6146c7b551e398a832f6da27879933 in 852ms, sequenceid=15, compaction requested=false 2023-03-23 19:55:39,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:39,181 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:39,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:39,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:39,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:39,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:39,184 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:39,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:39,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:39,184 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:39,185 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,185 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:39,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:39,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:39,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:39,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:39,189 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:39,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:39,190 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:39,190 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:39,191 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:39,191 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:39,191 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:39,344 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-23 19:55:39,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:39,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:39,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601399431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601399432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601399433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601399433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601399434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601399537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601399540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601399540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601399540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601399541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,602 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/e247d9b60ec04f59bdb846d3da3bd6fb 2023-03-23 19:55:39,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601399740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601399749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601399750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601399750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:39,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:39,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601399750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,012 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/a3eecf50fac84bfab9202ef00f8c43eb 2023-03-23 19:55:40,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601400043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601400054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601400054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601400055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601400056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,421 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/7460afdfd49749c19a33e69eec0f0b07 2023-03-23 19:55:40,424 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/e247d9b60ec04f59bdb846d3da3bd6fb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb 2023-03-23 19:55:40,428 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb, entries=150, sequenceid=33, filesize=11.6 K 2023-03-23 19:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/a3eecf50fac84bfab9202ef00f8c43eb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb 2023-03-23 19:55:40,432 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb, entries=150, sequenceid=33, filesize=11.6 K 2023-03-23 19:55:40,433 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/7460afdfd49749c19a33e69eec0f0b07 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07 2023-03-23 19:55:40,436 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07, entries=150, sequenceid=33, filesize=11.6 K 2023-03-23 19:55:40,437 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6b6146c7b551e398a832f6da27879933 in 1247ms, sequenceid=33, compaction requested=false 2023-03-23 19:55:40,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:40,437 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:40,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:40,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:40,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,445 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70606b8b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:40,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:40,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:40,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:40,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:40,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,453 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:40,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:40,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:40,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:40,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:40,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:40,463 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,463 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,463 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:40,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,469 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:40,469 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:40,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:40,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:40,569 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:40,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:40,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:40,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:40,572 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:40,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:40,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:40,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,574 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:40,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:40,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:40,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:40,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:40,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:40,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:40,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4107240[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:40,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:40,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601400574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,584 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,584 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:40,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:40,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:40,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:40,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601400578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601400579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601400579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601400580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:40,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:40,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:40,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:40,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:40,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:40,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:40,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,596 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,597 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:40,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,601 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:40,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:40,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,601 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:40,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601400683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601400686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601400686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601400687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601400687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,701 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:40,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:40,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:40,705 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:40,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:40,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:40,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:40,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:40,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:40,710 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:40,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:40,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:40,715 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:40,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:40,718 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:40,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22bc2570[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:40,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:40,719 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:40,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:40,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:40,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:40,723 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:40,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:40,726 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:40,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:40,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:40,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:40,726 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:40,727 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,727 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,727 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:40,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:40,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:40,731 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,731 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:40,731 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:40,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,732 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,832 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:40,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:40,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:40,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:40,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:40,835 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:40,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:40,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:40,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:40,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,838 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:40,839 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,839 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:40,839 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:40,839 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:40,839 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:40,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:40,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f106e0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:40,843 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:40,843 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,844 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,845 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,845 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:40,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:40,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:40,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:40,846 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,846 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:40,848 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:40,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:40,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:40,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:40,848 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:40,850 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,850 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,850 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,850 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,851 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,854 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:40,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:40,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:40,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,855 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:40,855 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:40,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601400889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601400890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601400890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601400891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:40,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601400891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,955 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:40,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:40,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:40,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:40,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:40,960 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:40,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:40,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:40,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,962 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:40,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:40,963 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:40,964 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:40,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/a54e8c7683a843fc8e0a173979eac4d1 2023-03-23 19:55:40,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,965 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57be0dc6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:40,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:40,967 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:40,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:40,969 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,969 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:40,971 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,971 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:40,973 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:40,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:40,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:40,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:40,973 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:40,976 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,976 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:40,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,976 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:40,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:40,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:40,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:40,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:40,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:40,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:40,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:40,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:40,981 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:40,981 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:40,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:40,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:40,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:40,981 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,081 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,084 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,085 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,090 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50bbf703[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,094 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,095 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,097 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,097 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,097 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,103 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,104 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601401193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601401194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601401196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601401196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601401197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,204 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,206 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,206 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,210 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,210 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,210 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,210 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,212 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13efdfc1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,216 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,216 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,220 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,220 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,221 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,222 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,224 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,225 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:55:41,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,226 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,230 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,230 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,330 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,333 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,337 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,337 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,337 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,337 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f851d5a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,343 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,343 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,347 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,348 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,349 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,358 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,358 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,359 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,359 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/004c8966145649f1894522f4d7ec5abd 2023-03-23 19:55:41,459 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,462 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,466 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18f2e751[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,472 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,474 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,476 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,476 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,478 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,478 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,479 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,479 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,480 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,482 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,483 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,483 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,483 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,583 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,586 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,591 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,593 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,593 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17fe08a3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,599 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,599 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,599 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,599 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,599 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,602 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,603 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,605 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,605 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,606 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601401698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601401701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601401702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,706 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601401703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601401704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,710 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,714 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,714 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,714 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:41,714 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:41,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:41,715 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:41,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a82cc4a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:41,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:41,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:41,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:41,721 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:41,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:41,722 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,722 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:41,724 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:41,725 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:41,725 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:41,725 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:41,725 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:41,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:41,726 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:41,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,734 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:41,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:41,734 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:41,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:41,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,735 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:41,735 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:41,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:41,736 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/5dbebe92901e4a9bba74f7da22538064 2023-03-23 19:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/a54e8c7683a843fc8e0a173979eac4d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1 2023-03-23 19:55:41,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1, entries=200, sequenceid=48, filesize=13.9 K 2023-03-23 19:55:41,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/004c8966145649f1894522f4d7ec5abd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd 2023-03-23 19:55:41,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd, entries=150, sequenceid=48, filesize=11.6 K 2023-03-23 19:55:41,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/5dbebe92901e4a9bba74f7da22538064 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064 2023-03-23 19:55:41,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064, entries=150, sequenceid=48, filesize=11.6 K 2023-03-23 19:55:41,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6b6146c7b551e398a832f6da27879933 in 1247ms, sequenceid=48, compaction requested=true 2023-03-23 19:55:41,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:41,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:41,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:41,803 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:41,803 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:41,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:41,804 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:41,804 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:41,804 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:41,804 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,804 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:41,805 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=37.1 K 2023-03-23 19:55:41,805 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,805 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=34.8 K 2023-03-23 19:55:41,805 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 8c9734a094834c9bb84877d137fd2148, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601338271 2023-03-23 19:55:41,805 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting af73f36fd1f9406aa3acd2541fc1ec01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601338271 2023-03-23 19:55:41,805 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e247d9b60ec04f59bdb846d3da3bd6fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679601338285 2023-03-23 19:55:41,805 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a3eecf50fac84bfab9202ef00f8c43eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679601338285 2023-03-23 19:55:41,806 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a54e8c7683a843fc8e0a173979eac4d1, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339431 2023-03-23 19:55:41,806 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 004c8966145649f1894522f4d7ec5abd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339432 2023-03-23 19:55:41,812 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:41,812 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#375 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:41,835 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:41,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:41,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:41,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:41,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:41,838 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:41,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:41,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:41,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,840 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,840 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:41,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:41,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:41,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:41,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:41,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:41,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:41,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:41,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:41,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:41,842 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-23 19:55:41,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:41,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:41,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:41,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:41,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:41,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:41,850 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/4066b25c5633427194abd110fd87ef4b 2023-03-23 19:55:41,859 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/3a92188616474fa38a5f13eb80ebd8dc 2023-03-23 19:55:42,226 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/e6da374ffcc6408896b85f5ab2cb3155 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e6da374ffcc6408896b85f5ab2cb3155 2023-03-23 19:55:42,229 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/e888262019554526a2c9ca256a621077 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e888262019554526a2c9ca256a621077 2023-03-23 19:55:42,230 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into e6da374ffcc6408896b85f5ab2cb3155(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:42,230 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,231 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601341803; duration=0sec 2023-03-23 19:55:42,231 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:42,231 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:42,232 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:42,232 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:42,232 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,233 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=34.8 K 2023-03-23 19:55:42,233 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 689bedacd0334aff84cfb3f65a5ce14d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601338271 2023-03-23 19:55:42,233 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7460afdfd49749c19a33e69eec0f0b07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679601338285 2023-03-23 19:55:42,234 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5dbebe92901e4a9bba74f7da22538064, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339432 2023-03-23 19:55:42,234 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into e888262019554526a2c9ca256a621077(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:42,234 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,234 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601341803; duration=0sec 2023-03-23 19:55:42,234 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:42,239 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:42,269 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/588e065066fb46f8903fd34c6c216027 2023-03-23 19:55:42,273 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/4066b25c5633427194abd110fd87ef4b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b 2023-03-23 19:55:42,278 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b, entries=150, sequenceid=65, filesize=11.6 K 2023-03-23 19:55:42,279 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/3a92188616474fa38a5f13eb80ebd8dc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc 2023-03-23 19:55:42,281 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc, entries=150, sequenceid=65, filesize=11.6 K 2023-03-23 19:55:42,282 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/588e065066fb46f8903fd34c6c216027 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027 2023-03-23 19:55:42,286 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027, entries=150, sequenceid=65, filesize=11.6 K 2023-03-23 19:55:42,286 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 6b6146c7b551e398a832f6da27879933 in 444ms, sequenceid=65, compaction requested=false 2023-03-23 19:55:42,287 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,287 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,287 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d7a618d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,292 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,293 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,294 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,297 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,297 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,299 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,300 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,304 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,303 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,304 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,304 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,404 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:42,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:42,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:42,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:42,407 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:42,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:42,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:42,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,418 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,418 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,419 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:42,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:42,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:42,421 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,421 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:42,421 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,421 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,423 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,423 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623ddb1c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,427 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,429 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,433 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,433 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,433 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,436 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,439 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,439 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,439 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,440 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:42,540 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:42,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:42,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:42,542 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:42,542 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:42,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:42,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:42,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:42,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:42,546 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,546 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:42,547 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,547 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,548 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5617ce8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,553 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,554 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,557 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,557 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,560 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,570 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,570 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,570 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,667 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/28be63099956421a885bb38d8417315e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28be63099956421a885bb38d8417315e 2023-03-23 19:55:42,670 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:42,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:42,672 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 28be63099956421a885bb38d8417315e(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:42,672 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,672 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601341803; duration=0sec 2023-03-23 19:55:42,672 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:42,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:42,674 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:42,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:42,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:42,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,685 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:42,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:42,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:42,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:42,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:42,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:42,688 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:42,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:42,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:42,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,690 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,691 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,692 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ed1092[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,694 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,699 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,699 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,703 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,706 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,706 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,706 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,707 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,706 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:42,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:42,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:55:42,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:42,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:42,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:42,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:42,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:42,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:42,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601402732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601402737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601402740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601402740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601402740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,807 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:42,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:42,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:42,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:42,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:42,810 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:42,810 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,810 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:42,810 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:42,810 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,811 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,812 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:42,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:42,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:42,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:42,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:42,813 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:42,813 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,814 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:42,814 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:42,814 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,814 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,816 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57a5722b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,824 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,824 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:42,831 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,831 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601402841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601402841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601402848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601402848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:42,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601402848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,932 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:42,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:42,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:42,937 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:42,937 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:42,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:42,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,938 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,939 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:42,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:42,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:42,941 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:42,942 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,942 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:42,942 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:42,943 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bccb6f8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:42,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:42,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:42,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:42,949 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:42,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:42,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:42,952 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:42,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:42,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:42,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:42,952 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:42,953 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,953 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,953 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:42,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:42,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:42,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:42,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:42,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:42,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:42,957 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:42,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:42,957 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:42,957 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:42,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601403046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601403046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601403052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601403054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,058 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:43,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601403054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:43,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:43,061 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:43,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:43,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:43,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,063 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:43,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:43,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:43,065 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,065 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:43,065 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:43,065 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:43,070 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:43,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c40755[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:43,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:43,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:43,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:43,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:43,077 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:43,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:43,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:43,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:43,077 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:43,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,080 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,081 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,082 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:43,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:43,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,084 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,085 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:43,085 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:43,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/ffe79afef80e441e9dd8100a5067906f 2023-03-23 19:55:43,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/d9f16eb524e347d4959bddfb45ce2f54 2023-03-23 19:55:43,185 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:43,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:43,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:43,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:43,188 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:43,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:43,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:43,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:43,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:43,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,194 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542cf4ff[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:43,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:43,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,198 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:43,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:43,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:43,201 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:43,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:43,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:43,201 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:43,201 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:43,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,206 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:43,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,209 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:43,209 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:43,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:43,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,309 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:43,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:43,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:43,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:43,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:43,312 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:43,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,313 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:43,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:43,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,316 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:43,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:43,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:43,318 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e4e07b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:43,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:43,320 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:43,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:43,324 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:43,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:43,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:43,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:43,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:43,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:43,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:43,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,337 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,339 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:43,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,340 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:43,340 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:43,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601403350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601403354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601403358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601403358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601403360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,440 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:43,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:43,443 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:43,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:43,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:43,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:43,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:43,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:43,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:43,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:43,447 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:43,448 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a1f8e9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:43,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:43,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:43,452 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,452 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:43,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:43,458 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:43,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:43,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:43,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:43,459 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:43,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:43,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:43,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,465 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:43,465 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:43,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:43,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:43,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c62da57c88f04d20a17fe22a95ab1344 2023-03-23 19:55:43,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/ffe79afef80e441e9dd8100a5067906f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f 2023-03-23 19:55:43,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f, entries=250, sequenceid=83, filesize=16.2 K 2023-03-23 19:55:43,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/d9f16eb524e347d4959bddfb45ce2f54 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54 2023-03-23 19:55:43,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54, entries=150, sequenceid=83, filesize=11.6 K 2023-03-23 19:55:43,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c62da57c88f04d20a17fe22a95ab1344 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344 2023-03-23 19:55:43,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344, entries=150, sequenceid=83, filesize=11.6 K 2023-03-23 19:55:43,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6b6146c7b551e398a832f6da27879933 in 846ms, sequenceid=83, compaction requested=true 2023-03-23 19:55:43,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:43,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:43,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-23 19:55:43,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-23 19:55:43,559 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:43,559 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:43,560 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:43,560 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:43,560 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:43,560 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:43,561 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e888262019554526a2c9ca256a621077, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=39.5 K 2023-03-23 19:55:43,561 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,561 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28be63099956421a885bb38d8417315e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=34.9 K 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e888262019554526a2c9ca256a621077, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339432 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 28be63099956421a885bb38d8417315e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339432 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4066b25c5633427194abd110fd87ef4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679601340577 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 588e065066fb46f8903fd34c6c216027, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679601340577 2023-03-23 19:55:43,561 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting ffe79afef80e441e9dd8100a5067906f, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342710 2023-03-23 19:55:43,562 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c62da57c88f04d20a17fe22a95ab1344, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342712 2023-03-23 19:55:43,565 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:43,567 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#385 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:43,568 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:43,573 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:43,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:43,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:43,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,576 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:43,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:43,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:43,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:43,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:43,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:43,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:43,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:43,578 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:43,578 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:43,586 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/b1ddb2fcf2d24d24a6d25b56765e1a8b 2023-03-23 19:55:43,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:43,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:43,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601403868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601403873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601403874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601403875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601403875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,976 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/cb5d240cb4234385ae8dfe82fe711e03 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cb5d240cb4234385ae8dfe82fe711e03 2023-03-23 19:55:43,980 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/1dc4e66e41c948d8bd2437eff8949ca6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/1dc4e66e41c948d8bd2437eff8949ca6 2023-03-23 19:55:43,984 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into cb5d240cb4234385ae8dfe82fe711e03(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:43,984 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:43,984 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601343559; duration=0sec 2023-03-23 19:55:43,984 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:43,984 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601403981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601403981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,985 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 1dc4e66e41c948d8bd2437eff8949ca6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,985 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601403981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,985 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601343559; duration=0sec 2023-03-23 19:55:43,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:43,985 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:43,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:43,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601403982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:43,986 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:43,986 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e6da374ffcc6408896b85f5ab2cb3155, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=34.9 K 2023-03-23 19:55:43,986 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e6da374ffcc6408896b85f5ab2cb3155, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679601339432 2023-03-23 19:55:43,986 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3a92188616474fa38a5f13eb80ebd8dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679601340577 2023-03-23 19:55:43,987 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d9f16eb524e347d4959bddfb45ce2f54, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342712 2023-03-23 19:55:43,991 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:44,004 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/c8ee0a7d4ba1488892038b43e0109a77 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c8ee0a7d4ba1488892038b43e0109a77 2023-03-23 19:55:44,005 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/fb7ed1900513437eb3c0444181478361 2023-03-23 19:55:44,009 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into c8ee0a7d4ba1488892038b43e0109a77(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:44,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:44,009 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601343559; duration=0sec 2023-03-23 19:55:44,009 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:44,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601404186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601404186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601404187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601404187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,418 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/a272081c20b748048740aaf22ad266ca 2023-03-23 19:55:44,422 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/b1ddb2fcf2d24d24a6d25b56765e1a8b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b 2023-03-23 19:55:44,425 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b, entries=150, sequenceid=103, filesize=11.6 K 2023-03-23 19:55:44,426 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/fb7ed1900513437eb3c0444181478361 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361 2023-03-23 19:55:44,429 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361, entries=150, sequenceid=103, filesize=11.6 K 2023-03-23 19:55:44,429 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/a272081c20b748048740aaf22ad266ca as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca 2023-03-23 19:55:44,434 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca, entries=150, sequenceid=103, filesize=11.6 K 2023-03-23 19:55:44,434 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6b6146c7b551e398a832f6da27879933 in 856ms, sequenceid=103, compaction requested=false 2023-03-23 19:55:44,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:44,435 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:44,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:44,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,437 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:44,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63aabbc7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:44,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:44,438 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:44,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:44,441 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:44,444 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:44,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:44,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:44,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:44,444 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:44,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,447 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,450 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,452 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:44,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,453 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,453 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:44,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,453 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:44,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:44,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:44,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:44,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/53170059e08c49eaacf8c76f00e014df 2023-03-23 19:55:44,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/8b32bcc897d347aaac563ffa96ea6912 2023-03-23 19:55:44,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601404518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601404519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601404519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601404520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,553 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:44,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:44,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:44,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:44,556 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:44,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:44,556 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:44,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,559 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:44,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:44,560 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:44,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:44,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:44,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:44,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50123041[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:44,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,565 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:44,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:44,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:44,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:44,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:44,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:44,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:44,569 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:44,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,574 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:44,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:44,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,575 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:44,575 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,575 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601404623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601404626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601404626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601404628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,675 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:44,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:44,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:44,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:44,679 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:44,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:44,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:44,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,681 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:44,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:44,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:44,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:44,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:44,683 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:44,685 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,685 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7ae500[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:44,686 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:44,686 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:44,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:44,687 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:44,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:44,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:44,691 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:44,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:44,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:44,691 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:44,691 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:44,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:44,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:44,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,700 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:44,700 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,700 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,801 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:44,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:44,804 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:44,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:44,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:44,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,804 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:44,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:44,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:44,808 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:44,808 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,808 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:44,808 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:44,808 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:44,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,811 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468528b6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:44,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:44,812 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,815 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,815 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:44,815 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:44,815 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:44,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:44,816 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:44,819 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:44,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:44,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:44,819 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:44,819 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,822 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,822 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:44,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:44,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,826 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:44,827 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:44,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601404826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601404831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601404831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601404832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601404879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/735e192b5f474d01bb3aab4dc9b287ca 2023-03-23 19:55:44,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/53170059e08c49eaacf8c76f00e014df as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df 2023-03-23 19:55:44,927 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:44,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:44,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:44,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df, entries=250, sequenceid=120, filesize=16.2 K 2023-03-23 19:55:44,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/8b32bcc897d347aaac563ffa96ea6912 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912 2023-03-23 19:55:44,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:44,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:44,930 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:44,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:44,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:44,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,930 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,933 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912, entries=150, sequenceid=120, filesize=11.6 K 2023-03-23 19:55:44,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/735e192b5f474d01bb3aab4dc9b287ca as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca 2023-03-23 19:55:44,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,934 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:44,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:44,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:44,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:44,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:44,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:44,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:44,935 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:44,936 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,936 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:44,936 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:44,936 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:44,936 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:44,937 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,937 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,938 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,938 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca, entries=150, sequenceid=120, filesize=11.6 K 2023-03-23 19:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:44,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6b6146c7b551e398a832f6da27879933 in 444ms, sequenceid=120, compaction requested=true 2023-03-23 19:55:44,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:44,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c7d722[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:44,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:44,939 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:44,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:44,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:44,939 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:44,940 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:44,940 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:44,940 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:44,941 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:44,941 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,941 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:44,941 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/1dc4e66e41c948d8bd2437eff8949ca6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=39.6 K 2023-03-23 19:55:44,941 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c8ee0a7d4ba1488892038b43e0109a77, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=35.0 K 2023-03-23 19:55:44,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,941 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c8ee0a7d4ba1488892038b43e0109a77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342712 2023-03-23 19:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:44,941 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 1dc4e66e41c948d8bd2437eff8949ca6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342712 2023-03-23 19:55:44,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:44,941 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:44,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:44,942 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting fb7ed1900513437eb3c0444181478361, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679601342738 2023-03-23 19:55:44,942 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b1ddb2fcf2d24d24a6d25b56765e1a8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679601342738 2023-03-23 19:55:44,943 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:44,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,943 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8b32bcc897d347aaac563ffa96ea6912, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,944 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 53170059e08c49eaacf8c76f00e014df, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:44,947 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:44,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:44,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:44,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:44,947 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:44,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:44,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:44,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:44,951 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#393 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:44,952 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:44,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:44,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:44,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:44,960 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:44,961 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:44,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:44,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:45,061 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:45,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:45,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:45,064 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:45,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:45,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:45,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:45,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,066 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:45,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:45,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:45,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:45,067 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:45,067 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:45,067 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:45,068 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:45,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,075 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/00ae6a674f9845aba454aa43290b4af0 2023-03-23 19:55:45,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:45,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:45,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601405148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601405148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601405149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601405150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601405258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601405259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601405260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601405260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,367 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6f6fe2ef22034ed3bdbd337d6948d94e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6f6fe2ef22034ed3bdbd337d6948d94e 2023-03-23 19:55:45,367 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa 2023-03-23 19:55:45,371 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 6f6fe2ef22034ed3bdbd337d6948d94e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:45,371 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:45,371 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601344939; duration=0sec 2023-03-23 19:55:45,371 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:45,371 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:45,372 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into b2b62ebf6eda4ec2a8b01ab38b9ca6fa(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:45,372 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:45,372 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601344939; duration=0sec 2023-03-23 19:55:45,372 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:45,372 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:45,372 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:45,372 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:45,373 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cb5d240cb4234385ae8dfe82fe711e03, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=35.0 K 2023-03-23 19:55:45,373 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting cb5d240cb4234385ae8dfe82fe711e03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679601342712 2023-03-23 19:55:45,373 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a272081c20b748048740aaf22ad266ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679601342738 2023-03-23 19:55:45,374 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 735e192b5f474d01bb3aab4dc9b287ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:45,378 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:45,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601405464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601405464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601405465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601405466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,484 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/071e6e06fb174dd58b7578b41773bc71 2023-03-23 19:55:45,494 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/f11ec8d5131d4cceba7026a6371f13e4 2023-03-23 19:55:45,497 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/00ae6a674f9845aba454aa43290b4af0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0 2023-03-23 19:55:45,500 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0, entries=150, sequenceid=138, filesize=11.8 K 2023-03-23 19:55:45,501 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/071e6e06fb174dd58b7578b41773bc71 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71 2023-03-23 19:55:45,503 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71, entries=150, sequenceid=138, filesize=11.8 K 2023-03-23 19:55:45,504 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/f11ec8d5131d4cceba7026a6371f13e4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4 2023-03-23 19:55:45,508 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4, entries=150, sequenceid=138, filesize=11.8 K 2023-03-23 19:55:45,508 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6b6146c7b551e398a832f6da27879933 in 440ms, sequenceid=138, compaction requested=false 2023-03-23 19:55:45,508 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:45,508 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:45,511 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:45,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31967143[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:45,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:45,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,514 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:45,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:45,516 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:45,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:45,518 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:45,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:45,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:45,518 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:45,518 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:45,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:45,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:45,520 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:45,527 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:45,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:45,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:45,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,527 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:45,527 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:45,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:45,528 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,628 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:45,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:45,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:45,631 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,633 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:45,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:45,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:45,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:45,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:45,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:45,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:45,635 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:45,635 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:45,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:45,787 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/217af493e7114578b3e3775d771129b9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/217af493e7114578b3e3775d771129b9 2023-03-23 19:55:45,790 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 217af493e7114578b3e3775d771129b9(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:45,790 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:45,791 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601344939; duration=0sec 2023-03-23 19:55:45,791 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601405793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601405794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601405797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601405799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601405900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601405900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601405903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:45,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601405905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,041 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/b38e4d9f97f94e6288ad91fa97df8705 2023-03-23 19:55:46,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601406105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601406106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601406106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601406112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601406410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601406411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601406413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601406418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,451 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/6c6b64e724df40f2a496eb8429cec3d1 2023-03-23 19:55:46,860 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/cfb5404bc5454f7293c105214c1011ca 2023-03-23 19:55:46,863 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/b38e4d9f97f94e6288ad91fa97df8705 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705 2023-03-23 19:55:46,868 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705, entries=150, sequenceid=154, filesize=11.8 K 2023-03-23 19:55:46,868 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/6c6b64e724df40f2a496eb8429cec3d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1 2023-03-23 19:55:46,877 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1, entries=150, sequenceid=154, filesize=11.8 K 2023-03-23 19:55:46,878 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/cfb5404bc5454f7293c105214c1011ca as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca 2023-03-23 19:55:46,881 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca, entries=150, sequenceid=154, filesize=11.8 K 2023-03-23 19:55:46,882 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6b6146c7b551e398a832f6da27879933 in 1247ms, sequenceid=154, compaction requested=true 2023-03-23 19:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:46,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ccd094[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:46,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:46,886 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,887 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:46,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:46,889 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:46,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:46,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:46,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:46,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:46,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:46,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:46,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:46,896 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,900 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,903 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:46,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:46,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:46,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:46,903 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:46,903 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:46,904 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:46,904 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:46,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:46,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:46,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601406922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601406927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601406928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601406928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:46,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601406929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,003 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,006 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,008 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,008 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,010 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56639f83[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,016 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,017 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,017 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,018 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,018 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,020 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,021 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,021 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,022 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,023 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,027 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,027 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,027 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,027 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:47,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601407030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601407038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601407039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601407039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,127 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,130 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,133 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,134 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,134 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,134 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18533caa[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,141 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,144 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,144 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,144 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,145 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,145 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,145 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,146 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,147 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,150 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,150 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,150 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,150 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,151 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:47,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601407235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601407243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601407243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601407243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,250 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,254 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,260 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,262 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,262 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,263 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a4d5a8a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,266 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,267 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,267 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,267 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,267 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,271 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,271 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,274 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,275 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,276 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,278 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,279 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,279 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/42e01d3defcc4f9890ea6fd809b4b20b 2023-03-23 19:55:47,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/cb36edfeeaa04c5d9d46d76f3142d3cd 2023-03-23 19:55:47,379 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,383 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,384 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,384 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,385 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,386 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,386 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,386 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,387 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45da7127[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,392 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,397 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,397 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,397 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,398 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,398 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,398 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,399 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,400 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,402 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,402 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,403 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,403 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,503 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,507 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,509 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,509 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,510 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,511 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,511 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,511 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,511 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,513 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8f6198[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,514 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,516 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,516 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,517 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,519 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,519 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,519 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,523 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,526 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,526 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,527 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,526 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601407541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601407548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601407549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:47,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601407550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,627 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,630 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,633 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,633 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,633 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:47,633 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:47,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:47,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,639 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e38aaaf[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:47,641 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:47,641 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:47,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:47,645 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:47,648 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:47,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:47,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:47,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:47,649 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:47,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:47,650 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:47,652 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:47,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:47,655 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:47,655 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,655 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:47,655 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:47,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:47,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c9fed8839c3e4486a01669f024ffd18a 2023-03-23 19:55:47,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/42e01d3defcc4f9890ea6fd809b4b20b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b 2023-03-23 19:55:47,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b, entries=200, sequenceid=174, filesize=14.1 K 2023-03-23 19:55:47,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/cb36edfeeaa04c5d9d46d76f3142d3cd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd 2023-03-23 19:55:47,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd, entries=150, sequenceid=174, filesize=11.8 K 2023-03-23 19:55:47,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c9fed8839c3e4486a01669f024ffd18a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a 2023-03-23 19:55:47,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a, entries=150, sequenceid=174, filesize=11.8 K 2023-03-23 19:55:47,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b6146c7b551e398a832f6da27879933 in 858ms, sequenceid=174, compaction requested=true 2023-03-23 19:55:47,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:47,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:47,752 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:47,752 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:47,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:47,753 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48302 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:47,754 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,754 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,754 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6f6fe2ef22034ed3bdbd337d6948d94e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=49.5 K 2023-03-23 19:55:47,754 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=47.2 K 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6f6fe2ef22034ed3bdbd337d6948d94e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:47,754 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b2b62ebf6eda4ec2a8b01ab38b9ca6fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 00ae6a674f9845aba454aa43290b4af0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679601344517 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 071e6e06fb174dd58b7578b41773bc71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679601344517 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b38e4d9f97f94e6288ad91fa97df8705, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679601345146 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6c6b64e724df40f2a496eb8429cec3d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679601345146 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 42e01d3defcc4f9890ea6fd809b4b20b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345793 2023-03-23 19:55:47,755 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting cb36edfeeaa04c5d9d46d76f3142d3cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345796 2023-03-23 19:55:47,755 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:47,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:47,759 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:47,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:47,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:47,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,761 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:47,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:47,761 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:47,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:47,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:47,763 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#405 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:47,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:47,764 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:47,765 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:47,765 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:47,766 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:47,766 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:47,797 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6ae52678483a406c9586f749ab801d58 2023-03-23 19:55:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:47,935 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:48,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601408022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601408049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601408056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601408057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601408058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601408127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,178 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/c949a149cb5b4d17b30528dc2dc44da7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c949a149cb5b4d17b30528dc2dc44da7 2023-03-23 19:55:48,181 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into c949a149cb5b4d17b30528dc2dc44da7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:48,182 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:48,182 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=12, startTime=1679601347752; duration=0sec 2023-03-23 19:55:48,182 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:48,182 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:48,183 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48302 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:48,183 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:48,183 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,183 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/217af493e7114578b3e3775d771129b9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=47.2 K 2023-03-23 19:55:48,184 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 217af493e7114578b3e3775d771129b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679601343861 2023-03-23 19:55:48,184 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f11ec8d5131d4cceba7026a6371f13e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1679601344517 2023-03-23 19:55:48,184 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting cfb5404bc5454f7293c105214c1011ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679601345146 2023-03-23 19:55:48,185 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c9fed8839c3e4486a01669f024ffd18a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345796 2023-03-23 19:55:48,191 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/0d35ae88fd6f4bd19b1c1801fadf2e1f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/0d35ae88fd6f4bd19b1c1801fadf2e1f 2023-03-23 19:55:48,192 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#409 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:48,195 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 0d35ae88fd6f4bd19b1c1801fadf2e1f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:48,195 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:48,195 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=12, startTime=1679601347752; duration=0sec 2023-03-23 19:55:48,195 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:48,199 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/f50065ebc17c46ed8b997ea58c81d4bb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f50065ebc17c46ed8b997ea58c81d4bb 2023-03-23 19:55:48,202 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into f50065ebc17c46ed8b997ea58c81d4bb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:48,202 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:48,202 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=12, startTime=1679601347753; duration=0sec 2023-03-23 19:55:48,202 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:48,206 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/55247cecede3442391e0af2c89a10ffa 2023-03-23 19:55:48,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601408333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,614 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/bc58a819d463451394b8d2d73ef9c6dc 2023-03-23 19:55:48,617 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6ae52678483a406c9586f749ab801d58 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58 2023-03-23 19:55:48,620 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58, entries=150, sequenceid=187, filesize=11.8 K 2023-03-23 19:55:48,621 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/55247cecede3442391e0af2c89a10ffa as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa 2023-03-23 19:55:48,624 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa, entries=150, sequenceid=187, filesize=11.8 K 2023-03-23 19:55:48,625 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/bc58a819d463451394b8d2d73ef9c6dc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc 2023-03-23 19:55:48,628 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc, entries=150, sequenceid=187, filesize=11.8 K 2023-03-23 19:55:48,629 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b6146c7b551e398a832f6da27879933 in 864ms, sequenceid=187, compaction requested=false 2023-03-23 19:55:48,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:48,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:48,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5128e8[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:48,634 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:48,634 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,635 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:48,635 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,636 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:48,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:48,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:48,640 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:48,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:48,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:48,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:48,640 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:48,641 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:48,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:48,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,643 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,644 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,646 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:48,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:48,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:48,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,650 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:48,649 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,650 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,650 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:48,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6664250b9f6a44e289c8d1adeb190797 2023-03-23 19:55:48,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/8acedb5cb4344855a30b34dc9e6499be 2023-03-23 19:55:48,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601408686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,750 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:48,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:48,753 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:48,753 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:48,753 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:48,753 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:48,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,755 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:48,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:48,755 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,755 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:48,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:48,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:48,757 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,757 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:48,757 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:48,757 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:48,758 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68943d4c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:48,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:48,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,761 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:48,762 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,762 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:48,763 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:48,766 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:48,766 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:48,766 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:48,766 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:48,766 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:48,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,769 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:48,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,771 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:48,771 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:48,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35756 deadline: 1679601408795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,871 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:48,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:48,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:48,874 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:48,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:48,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:48,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:48,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,877 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:48,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:48,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:48,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:48,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:48,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:48,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:48,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:48,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,880 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce30267[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:48,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:48,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,883 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:48,883 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,883 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,888 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:48,890 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:48,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:48,891 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:48,891 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:48,891 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:48,892 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,892 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,892 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:48,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,893 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:48,895 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:48,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:48,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:48,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:48,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:48,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:48,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:48,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:48,897 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:48,897 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:48,997 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:48,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:48,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:49,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:49,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:49,000 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:49,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:49,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:49,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,002 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:49,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,002 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:49,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:49,004 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:49,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601409001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,004 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:49,004 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:49,004 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:49,007 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c5be86[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:49,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:49,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,011 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:49,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:49,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:49,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:49,015 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:49,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:49,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:49,015 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:49,015 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:49,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,017 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,017 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:49,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:49,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:49,021 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:49,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:49,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,022 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:49,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,022 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:49,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:49,022 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:49,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601409056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601409063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/35581c85a4fe4ee6a81c7863d1ec130f 2023-03-23 19:55:49,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601409066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6664250b9f6a44e289c8d1adeb190797 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797 2023-03-23 19:55:49,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601409072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797, entries=200, sequenceid=210, filesize=14.1 K 2023-03-23 19:55:49,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/8acedb5cb4344855a30b34dc9e6499be as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be 2023-03-23 19:55:49,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be, entries=150, sequenceid=210, filesize=11.8 K 2023-03-23 19:55:49,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/35581c85a4fe4ee6a81c7863d1ec130f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f 2023-03-23 19:55:49,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f, entries=150, sequenceid=210, filesize=11.8 K 2023-03-23 19:55:49,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6b6146c7b551e398a832f6da27879933 in 444ms, sequenceid=210, compaction requested=true 2023-03-23 19:55:49,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:49,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:49,085 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:49,085 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:49,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:49,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38942 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:49,086 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:49,086 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/0d35ae88fd6f4bd19b1c1801fadf2e1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=38.0 K 2023-03-23 19:55:49,086 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,086 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c949a149cb5b4d17b30528dc2dc44da7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=35.7 K 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 0d35ae88fd6f4bd19b1c1801fadf2e1f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345796 2023-03-23 19:55:49,086 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c949a149cb5b4d17b30528dc2dc44da7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345796 2023-03-23 19:55:49,087 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6ae52678483a406c9586f749ab801d58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679601346926 2023-03-23 19:55:49,087 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 55247cecede3442391e0af2c89a10ffa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679601346926 2023-03-23 19:55:49,087 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6664250b9f6a44e289c8d1adeb190797, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:49,087 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8acedb5cb4344855a30b34dc9e6499be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:49,092 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:49,093 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:49,105 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/abd25c3077c149ed955fdc74a133d82c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/abd25c3077c149ed955fdc74a133d82c 2023-03-23 19:55:49,108 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into abd25c3077c149ed955fdc74a133d82c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:49,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:49,108 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601349085; duration=0sec 2023-03-23 19:55:49,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:49,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:49,109 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:49,109 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:49,109 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,109 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f50065ebc17c46ed8b997ea58c81d4bb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=35.7 K 2023-03-23 19:55:49,110 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f50065ebc17c46ed8b997ea58c81d4bb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679601345796 2023-03-23 19:55:49,110 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bc58a819d463451394b8d2d73ef9c6dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679601346926 2023-03-23 19:55:49,110 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 35581c85a4fe4ee6a81c7863d1ec130f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:49,115 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:49,122 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:49,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:49,125 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,125 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:49,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:49,127 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,127 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:49,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:49,128 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,128 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:49,129 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,135 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/5ada4ba061944859a243310838d7f987 2023-03-23 19:55:49,145 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/bd3014aff6a04ee48a6d6f623610e5a9 2023-03-23 19:55:49,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:49,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:49,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601409407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,500 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/f9e1e1f8265d46ff96b9ab720ed11b99 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/f9e1e1f8265d46ff96b9ab720ed11b99 2023-03-23 19:55:49,504 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into f9e1e1f8265d46ff96b9ab720ed11b99(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:49,504 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:49,504 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601349085; duration=0sec 2023-03-23 19:55:49,504 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:49,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601409511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,523 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/eb1018997afa4ab7a9c261f723f61702 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/eb1018997afa4ab7a9c261f723f61702 2023-03-23 19:55:49,526 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into eb1018997afa4ab7a9c261f723f61702(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:49,526 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:49,526 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601349085; duration=0sec 2023-03-23 19:55:49,527 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:49,559 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/84bed03268c045c589fb1d06de031097 2023-03-23 19:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/5ada4ba061944859a243310838d7f987 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987 2023-03-23 19:55:49,566 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987, entries=150, sequenceid=223, filesize=11.8 K 2023-03-23 19:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/bd3014aff6a04ee48a6d6f623610e5a9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9 2023-03-23 19:55:49,570 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9, entries=150, sequenceid=223, filesize=11.8 K 2023-03-23 19:55:49,572 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/84bed03268c045c589fb1d06de031097 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097 2023-03-23 19:55:49,575 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097, entries=150, sequenceid=223, filesize=11.8 K 2023-03-23 19:55:49,576 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6b6146c7b551e398a832f6da27879933 in 448ms, sequenceid=223, compaction requested=false 2023-03-23 19:55:49,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:49,576 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2024cafe[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:49,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:49,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,581 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:49,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:49,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:49,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:49,587 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:49,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:49,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:49,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:49,587 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:49,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:49,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:49,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:49,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:49,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:49,593 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:49,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:49,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:49,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,593 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:49,593 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:49,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:49,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:49,694 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:49,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:49,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:49,696 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:49,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:49,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:49,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,697 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,699 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:49,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:49,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:49,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:49,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:49,700 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:49,701 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:49,701 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:49,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:49,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:49,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601409765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:49,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:49,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601409872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601410081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,109 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/c778b143d29b4650b3d2cd41bfd6b94c 2023-03-23 19:55:50,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:50,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601410388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,519 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/133f10aec2304127aa68eeeee26960b1 2023-03-23 19:55:50,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:50,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601410894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,928 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/f31fcdea2d7d41128aebbf505bd3f294 2023-03-23 19:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/c778b143d29b4650b3d2cd41bfd6b94c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c 2023-03-23 19:55:50,935 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c, entries=150, sequenceid=245, filesize=11.8 K 2023-03-23 19:55:50,936 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/133f10aec2304127aa68eeeee26960b1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1 2023-03-23 19:55:50,939 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1, entries=150, sequenceid=245, filesize=11.8 K 2023-03-23 19:55:50,939 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/f31fcdea2d7d41128aebbf505bd3f294 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294 2023-03-23 19:55:50,943 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294, entries=150, sequenceid=245, filesize=11.8 K 2023-03-23 19:55:50,943 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6b6146c7b551e398a832f6da27879933 in 1242ms, sequenceid=245, compaction requested=true 2023-03-23 19:55:50,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:50,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:50,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:50,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6913fdeb[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:50,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:50,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:50,950 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,950 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:50,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:50,952 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:50,952 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:50,955 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:50,955 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:50,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,960 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:50,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:50,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:50,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:50,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:50,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:50,965 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:50,965 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:50,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:50,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:50,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:51,065 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,069 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,071 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,071 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,071 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:51,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,072 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,073 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,073 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,073 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,073 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:51,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:51,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de72a94[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:51,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:51,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,079 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:51,079 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,079 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,081 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/d5ca18753d804afbaf21db88af5a2344 2023-03-23 19:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:51,084 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:51,084 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:51,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,087 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:51,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:51,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,091 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,091 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:51,092 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:51,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:51,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/c509d5035a294de5972d79c05eeadfa2 2023-03-23 19:55:51,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/adda84041dfe4d32a455ea90207cd1a0 2023-03-23 19:55:51,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/d5ca18753d804afbaf21db88af5a2344 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344 2023-03-23 19:55:51,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344, entries=250, sequenceid=259, filesize=16.5 K 2023-03-23 19:55:51,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/c509d5035a294de5972d79c05eeadfa2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2 2023-03-23 19:55:51,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2, entries=150, sequenceid=259, filesize=11.9 K 2023-03-23 19:55:51,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/adda84041dfe4d32a455ea90207cd1a0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0 2023-03-23 19:55:51,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0, entries=150, sequenceid=259, filesize=11.9 K 2023-03-23 19:55:51,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=93.93 KB/96180 for 6b6146c7b551e398a832f6da27879933 in 42ms, sequenceid=259, compaction requested=true 2023-03-23 19:55:51,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:51,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:51,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:51,114 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:51,114 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:51,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:51,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:51,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:51,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,116 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53570 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:51,117 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:51,117 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48790 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:51,117 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,117 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:51,117 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/f9e1e1f8265d46ff96b9ab720ed11b99, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=52.3 K 2023-03-23 19:55:51,117 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,117 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/abd25c3077c149ed955fdc74a133d82c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=47.6 K 2023-03-23 19:55:51,117 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f9e1e1f8265d46ff96b9ab720ed11b99, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:51,117 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting abd25c3077c149ed955fdc74a133d82c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5ada4ba061944859a243310838d7f987, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601348660 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting bd3014aff6a04ee48a6d6f623610e5a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601348660 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c778b143d29b4650b3d2cd41bfd6b94c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679601349382 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 133f10aec2304127aa68eeeee26960b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679601349382 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d5ca18753d804afbaf21db88af5a2344, keycount=250, bloomtype=ROW, size=16.5 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349734 2023-03-23 19:55:51,118 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c509d5035a294de5972d79c05eeadfa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349764 2023-03-23 19:55:51,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/bf8c1c79f027446a8a5c18deb352b0d9 2023-03-23 19:55:51,127 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:51,130 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#428 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:51,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601411136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601411136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601411140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601411144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,192 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,195 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,197 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,197 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,199 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,199 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,199 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,199 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:51,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:51,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ac8820[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:51,202 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:51,202 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,203 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:51,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:51,205 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:51,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:51,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:51,207 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:51,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:51,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:51,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:51,207 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:51,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,211 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:51,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:51,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:51,213 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:51,213 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,213 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601411245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601411246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601411246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601411255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,313 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,317 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,318 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,321 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:51,321 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:51,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1159d311[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:51,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:51,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,327 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:51,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:51,328 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:51,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:51,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:51,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:51,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:51,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:51,331 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:51,332 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,332 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,332 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,334 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:51,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,340 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:51,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:51,340 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:51,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,441 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,444 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,447 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:51,449 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67d38874[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:51,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:51,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:51,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,453 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:51,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:51,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:51,457 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:51,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:51,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:51,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:51,457 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:51,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601411452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601411452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601411453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:51,466 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,466 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:51,466 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:51,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601411461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,537 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/db00e784d5194dcbbd882e2328e5e147 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/db00e784d5194dcbbd882e2328e5e147 2023-03-23 19:55:51,541 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into db00e784d5194dcbbd882e2328e5e147(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:51,541 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:51,541 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=12, startTime=1679601351114; duration=0sec 2023-03-23 19:55:51,541 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:51,541 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:55:51,542 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48790 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:55:51,542 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:51,542 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,542 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/eb1018997afa4ab7a9c261f723f61702, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=47.6 K 2023-03-23 19:55:51,543 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting eb1018997afa4ab7a9c261f723f61702, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679601348005 2023-03-23 19:55:51,543 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 84bed03268c045c589fb1d06de031097, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679601348660 2023-03-23 19:55:51,543 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f31fcdea2d7d41128aebbf505bd3f294, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679601349382 2023-03-23 19:55:51,543 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting adda84041dfe4d32a455ea90207cd1a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349764 2023-03-23 19:55:51,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/ccd272f193a04119b8890a193fb92a15 2023-03-23 19:55:51,550 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/3d3ea6a34b9d47e2996002251e1598c7 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3d3ea6a34b9d47e2996002251e1598c7 2023-03-23 19:55:51,551 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:51,556 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into 3d3ea6a34b9d47e2996002251e1598c7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:51,556 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:51,556 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=12, startTime=1679601351114; duration=0sec 2023-03-23 19:55:51,556 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:51,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/49d480d08ce14c8cb3b3452c0d0a4916 2023-03-23 19:55:51,565 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/50c9f948e94a46f9923e45da0ec911ff as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/50c9f948e94a46f9923e45da0ec911ff 2023-03-23 19:55:51,566 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/bf8c1c79f027446a8a5c18deb352b0d9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9 2023-03-23 19:55:51,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,571 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,572 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 50c9f948e94a46f9923e45da0ec911ff(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:51,572 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:51,572 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=12, startTime=1679601351114; duration=0sec 2023-03-23 19:55:51,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9, entries=250, sequenceid=278, filesize=16.7 K 2023-03-23 19:55:51,572 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:51,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/ccd272f193a04119b8890a193fb92a15 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15 2023-03-23 19:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:51,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15, entries=150, sequenceid=278, filesize=11.9 K 2023-03-23 19:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/49d480d08ce14c8cb3b3452c0d0a4916 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916 2023-03-23 19:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b4bd90[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:51,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:51,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916, entries=150, sequenceid=278, filesize=11.9 K 2023-03-23 19:55:51,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:51,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:51,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:51,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:51,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b6146c7b551e398a832f6da27879933 in 467ms, sequenceid=278, compaction requested=false 2023-03-23 19:55:51,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:51,593 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,595 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:51,596 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,596 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:51,598 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:51,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:51,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:51,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:51,598 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:51,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:51,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,600 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:51,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:51,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:51,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:51,605 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:51,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:51,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:51,606 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:51,606 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:51,612 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,706 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:51,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:51,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:51,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:51,709 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:51,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:51,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:51,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:51,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:51,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:51,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:51,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:51,713 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:51,713 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:51,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:51,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601411786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601411786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601411802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601411802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601411903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601411903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601411908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601411908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:51,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601411910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601412106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601412107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35812 deadline: 1679601412113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601412113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,120 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/c1ac95630c5f427dbcda631208fa2d16 2023-03-23 19:55:52,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601412411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35766 deadline: 1679601412411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601412417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35840 deadline: 1679601412418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,529 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/e84c6cccd331433bb1eea36a03006854 2023-03-23 19:55:52,540 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/3e4bbbca492b4a25b2469aed1bac0621 2023-03-23 19:55:52,544 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/c1ac95630c5f427dbcda631208fa2d16 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16 2023-03-23 19:55:52,547 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16, entries=150, sequenceid=294, filesize=11.9 K 2023-03-23 19:55:52,547 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/e84c6cccd331433bb1eea36a03006854 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854 2023-03-23 19:55:52,552 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854, entries=150, sequenceid=294, filesize=11.9 K 2023-03-23 19:55:52,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/3e4bbbca492b4a25b2469aed1bac0621 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621 2023-03-23 19:55:52,556 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621, entries=150, sequenceid=294, filesize=11.9 K 2023-03-23 19:55:52,556 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b6146c7b551e398a832f6da27879933 in 843ms, sequenceid=294, compaction requested=true 2023-03-23 19:55:52,556 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:52,556 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:52,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40aace4e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:52,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:52,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,562 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:52,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:52,569 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:52,571 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:52,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:52,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:52,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:52,571 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:52,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:52,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,577 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:52,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:52,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:52,577 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:52,578 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:52,578 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:52,578 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:52,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:52,678 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:52,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:52,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:52,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:52,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:52,681 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:52,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:52,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,682 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:52,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,687 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:52,687 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,688 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:52,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:52,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:52,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:52,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:52,689 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:52,689 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:52,689 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:52,690 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:52,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601412930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601412934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601412935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:52,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:52,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601412935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601413037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601413037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601413041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601413041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,096 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/7201b42667474e4384838b6ce9db9a45 2023-03-23 19:55:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601413243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601413243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601413246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601413247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,509 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/610c7bc0c9024f34b5d510baff26c60e 2023-03-23 19:55:53,519 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/b5a9d4971313464885a126d3bb7ef9c3 2023-03-23 19:55:53,522 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/7201b42667474e4384838b6ce9db9a45 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45 2023-03-23 19:55:53,526 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45, entries=150, sequenceid=313, filesize=11.9 K 2023-03-23 19:55:53,527 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/610c7bc0c9024f34b5d510baff26c60e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e 2023-03-23 19:55:53,530 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e, entries=150, sequenceid=313, filesize=11.9 K 2023-03-23 19:55:53,531 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/b5a9d4971313464885a126d3bb7ef9c3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3 2023-03-23 19:55:53,534 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3, entries=150, sequenceid=313, filesize=11.9 K 2023-03-23 19:55:53,535 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b6146c7b551e398a832f6da27879933 in 846ms, sequenceid=313, compaction requested=true 2023-03-23 19:55:53,535 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:53,535 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:53,535 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:53,537 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:53,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb45c11[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:53,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:53,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:53,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:53,543 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:53,545 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:53,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:53,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:53,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:53,545 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:53,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,548 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,551 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,551 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:53,552 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:53,553 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:53,553 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:53,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:53,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:53,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601413581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601413584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601413588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601413589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,653 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:53,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:53,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:53,656 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:53,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,656 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:53,657 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:53,657 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,657 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,658 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,658 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,659 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,659 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:53,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:53,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:53,660 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,660 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:53,660 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:53,660 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:53,662 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:53,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dff7bb[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:53,664 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:53,664 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,669 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:53,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:53,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:53,671 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:53,673 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:53,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:53,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:53,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:53,673 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:53,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:53,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:53,679 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:53,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,679 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:53,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,679 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,680 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,680 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601413694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601413694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601413696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601413697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,779 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:53,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:53,782 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:53,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:53,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:53,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,785 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:53,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:53,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:53,786 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,788 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72919e8d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:53,791 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,792 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:53,792 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,792 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:53,793 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:53,795 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:53,795 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:53,795 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:53,796 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:53,796 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:53,801 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,801 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,801 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,801 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,804 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,804 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,805 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,805 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601413899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601413899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601413901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,905 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:53,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:53,908 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:53,908 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,908 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:53,908 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:53,909 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,909 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,910 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:53,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:53,912 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,912 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:53,912 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:53,912 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:53,912 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:53,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601413905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,918 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:53,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645ff62f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,920 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:53,921 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,921 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:53,922 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:53,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:53,924 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:53,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:53,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:53,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:53,925 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:53,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,926 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:53,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,926 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:53,927 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:53,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601413924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,929 DEBUG [Thread-2721] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:53,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:53,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:53,932 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:53,932 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:53,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:53,932 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:53,932 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:53,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/75cd8368d6d14c4c9caacad3da5511f9 2023-03-23 19:55:54,033 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,035 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,036 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,038 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,038 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,038 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,038 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,042 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59367a61[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,044 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,044 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,055 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,055 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,055 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,057 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,057 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,058 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,061 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,061 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,061 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,061 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,161 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,173 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,177 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,177 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,177 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,177 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cf3cd3[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,182 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,186 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,186 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,186 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,186 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,186 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,187 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,188 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,192 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,192 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601414205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601414205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601414205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601414214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,292 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,295 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,296 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,299 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,299 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,299 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,300 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,301 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c54b7f1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,304 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,313 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,313 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,313 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,313 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,313 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,315 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,316 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:54,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,320 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,320 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/dd63fd4b686a467b82e36c39d55f3d2f 2023-03-23 19:55:54,420 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,423 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,423 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,423 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,425 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,427 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,427 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,430 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2721d0da[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,437 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,437 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,439 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:54,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,444 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,445 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,545 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,548 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,550 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,550 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,552 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,554 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,554 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d2e532d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,555 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,555 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,557 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,557 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,559 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,561 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,561 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,564 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,564 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,568 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:54,568 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,668 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,671 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,671 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,672 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,672 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,673 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:54,674 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,674 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:54,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,677 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bcd7ff[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:54,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:54,680 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,681 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:54,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,682 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:54,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:54,685 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:54,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:54,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:54,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:54,685 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:54,694 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,694 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,694 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:54,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,698 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:54,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:54,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:54,698 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:54,698 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:54,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:54,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601414715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601414716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601414717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:54,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601414723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/da3763ad3e254d42965610b16707436c 2023-03-23 19:55:54,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/75cd8368d6d14c4c9caacad3da5511f9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9 2023-03-23 19:55:54,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9, entries=150, sequenceid=328, filesize=11.9 K 2023-03-23 19:55:54,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/dd63fd4b686a467b82e36c39d55f3d2f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f 2023-03-23 19:55:54,798 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:54,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f, entries=150, sequenceid=328, filesize=11.9 K 2023-03-23 19:55:54,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:54,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:54,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:54,802 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:54,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:54,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:54,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/da3763ad3e254d42965610b16707436c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c 2023-03-23 19:55:54,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,804 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:54,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:54,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:54,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:54,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:54,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:54,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c, entries=150, sequenceid=328, filesize=11.9 K 2023-03-23 19:55:54,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:54,809 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6b6146c7b551e398a832f6da27879933 in 1256ms, sequenceid=328, compaction requested=true 2023-03-23 19:55:54,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:54,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:54,809 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:55:54,809 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:55:54,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:54,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:54,810 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:55:54,810 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:54,810 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61562 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:55:54,810 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,810 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:54,810 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/db00e784d5194dcbbd882e2328e5e147, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=64.9 K 2023-03-23 19:55:54,810 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,811 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3d3ea6a34b9d47e2996002251e1598c7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=60.1 K 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting db00e784d5194dcbbd882e2328e5e147, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349764 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3d3ea6a34b9d47e2996002251e1598c7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349764 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting bf8c1c79f027446a8a5c18deb352b0d9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601351090 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ccd272f193a04119b8890a193fb92a15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601351090 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting c1ac95630c5f427dbcda631208fa2d16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1679601351135 2023-03-23 19:55:54,811 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting e84c6cccd331433bb1eea36a03006854, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1679601351135 2023-03-23 19:55:54,812 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7201b42667474e4384838b6ce9db9a45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679601351776 2023-03-23 19:55:54,812 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 610c7bc0c9024f34b5d510baff26c60e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679601351776 2023-03-23 19:55:54,812 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 75cd8368d6d14c4c9caacad3da5511f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:54,812 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting dd63fd4b686a467b82e36c39d55f3d2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:54,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:54,820 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:54,822 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:54,822 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:54,830 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#441 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:54,830 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#442 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:54,853 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/4d07ec5100b94fee90b66df345b0c138 2023-03-23 19:55:54,869 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/80b4d9509edc42d49a2c5cfc353239ad 2023-03-23 19:55:55,243 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/5ac20451ce814fa8955463d1cb0a34e2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5ac20451ce814fa8955463d1cb0a34e2 2023-03-23 19:55:55,248 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into 5ac20451ce814fa8955463d1cb0a34e2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:55,248 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,248 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=11, startTime=1679601354809; duration=0sec 2023-03-23 19:55:55,248 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:55,248 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:55:55,249 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61562 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:55:55,249 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:55,249 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,249 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/50c9f948e94a46f9923e45da0ec911ff, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=60.1 K 2023-03-23 19:55:55,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 50c9f948e94a46f9923e45da0ec911ff, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679601349764 2023-03-23 19:55:55,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 49d480d08ce14c8cb3b3452c0d0a4916, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601351090 2023-03-23 19:55:55,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3e4bbbca492b4a25b2469aed1bac0621, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1679601351135 2023-03-23 19:55:55,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b5a9d4971313464885a126d3bb7ef9c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679601351776 2023-03-23 19:55:55,251 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting da3763ad3e254d42965610b16707436c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:55,255 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/3fda712070954bc190cb1094159ce903 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/3fda712070954bc190cb1094159ce903 2023-03-23 19:55:55,258 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 3fda712070954bc190cb1094159ce903(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:55,258 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,258 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=11, startTime=1679601354809; duration=0sec 2023-03-23 19:55:55,258 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:55,259 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#446 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:55,281 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/28c7adf95455490083a2c31b21a8ee93 2023-03-23 19:55:55,284 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/4d07ec5100b94fee90b66df345b0c138 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138 2023-03-23 19:55:55,287 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138, entries=150, sequenceid=345, filesize=11.9 K 2023-03-23 19:55:55,287 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/80b4d9509edc42d49a2c5cfc353239ad as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad 2023-03-23 19:55:55,290 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad, entries=150, sequenceid=345, filesize=11.9 K 2023-03-23 19:55:55,291 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/28c7adf95455490083a2c31b21a8ee93 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93 2023-03-23 19:55:55,294 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93, entries=150, sequenceid=345, filesize=11.9 K 2023-03-23 19:55:55,294 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 6b6146c7b551e398a832f6da27879933 in 474ms, sequenceid=345, compaction requested=false 2023-03-23 19:55:55,294 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,294 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,296 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fce1de6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,298 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,299 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,300 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,301 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,302 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,304 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,304 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,306 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,311 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,311 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,411 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:55,414 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:55,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:55,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:55,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,414 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,417 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,417 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:55,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:55,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:55,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:55,418 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,420 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c0b7c5[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,422 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,424 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,424 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,428 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,428 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,428 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,430 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,431 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,432 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,435 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,434 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,435 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,435 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,435 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,535 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:55,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:55,538 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:55,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:55,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:55,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,539 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,540 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:55,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:55,542 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,542 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:55,542 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,542 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,542 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,544 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7948230b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,545 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,548 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,548 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,549 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,551 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,551 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,553 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,553 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,553 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,553 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,554 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,561 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,561 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,561 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,561 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,662 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:55,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:55,665 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:55,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:55,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:55,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,666 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,667 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,667 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,668 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/95553993607240d1beab6636ea54a018 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/95553993607240d1beab6636ea54a018 2023-03-23 19:55:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:55,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:55,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,672 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,672 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 95553993607240d1beab6636ea54a018(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,672 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,672 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=11, startTime=1679601354809; duration=0sec 2023-03-23 19:55:55,672 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695b10f2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,674 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,675 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,681 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,681 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,681 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,683 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,683 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,685 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,688 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,688 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,689 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,689 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,689 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:55,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:55,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:55,789 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601415785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35782 deadline: 1679601415785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601415785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601355749, max timestamp=1679601355790Num uniques 859; 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:55,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601355749, max timestamp=1679601355790Num uniques 859; 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601355749, max timestamp=1679601355790Num uniques 859; 2023-03-23 19:55:55,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:55:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:55,793 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,793 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,794 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,795 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,795 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,795 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:55,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,796 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,799 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,799 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76650112[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,801 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,801 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,801 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,802 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,803 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,805 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,805 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,805 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,807 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,808 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,809 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,813 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,813 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,813 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,813 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601415879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601415892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601415892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601415892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,913 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:55,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:55,916 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:55,916 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,916 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:55,916 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:55,917 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,917 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,918 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,918 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:55,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:55,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:55,920 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,920 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:55,920 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:55,920 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,921 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:55,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d8fe84[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:55,923 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,924 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:55,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,924 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:55,927 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:55,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:55,929 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:55,929 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:55,929 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:55,929 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:55,929 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:55,930 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,931 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,931 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,933 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:55,935 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:55,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:55,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:55,935 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:55,936 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:55,936 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:55,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:55,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:55,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601415985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,036 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,039 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,039 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,039 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,039 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,039 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,040 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,045 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,045 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,046 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,046 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:56,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,047 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,047 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,047 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,047 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:56,050 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d30d34f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:56,052 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:56,052 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,053 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:56,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,060 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:56,063 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:56,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:56,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:56,063 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:56,063 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:56,064 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,064 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,064 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,066 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:56,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:56,069 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,069 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:56,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601416098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601416098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601416099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/2f8d9d5f16ff40909d9d4018cb4d1f94 2023-03-23 19:55:56,169 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,173 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,173 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,174 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,175 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,175 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,176 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:56,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:56,182 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cdd9e9b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:56,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:56,183 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:56,185 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,185 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:56,188 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:56,191 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:56,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:56,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:56,191 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:56,191 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:56,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,193 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601416190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,196 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,198 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,199 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:56,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,199 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:56,199 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:56,299 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,303 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,303 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,304 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:56,309 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:56,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:56,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626e7583[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:56,311 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:56,311 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,312 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:56,313 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,313 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:56,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:56,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:56,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:56,315 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:56,315 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:56,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,317 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,317 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,318 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,322 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:56,322 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:56,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601416406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601416406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601416407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,422 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,425 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,425 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,425 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,425 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,426 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,427 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,427 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,429 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,429 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,429 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,429 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:56,433 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b08b94[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:56,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:56,434 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,436 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:56,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,437 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:56,438 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:56,440 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:56,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:56,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:56,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:56,440 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:56,441 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,441 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,441 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:56,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,445 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:56,445 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:56,445 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:56,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601416499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,545 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,548 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,549 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,551 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,551 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,552 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,553 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:56,555 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,555 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b119866[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:56,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:56,557 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,558 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:56,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:56,558 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:56,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,558 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:56,559 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,560 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/33184b7de1e14dcba204a2f2228950b1 2023-03-23 19:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:56,562 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:56,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:56,562 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:56,563 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:56,563 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:56,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,567 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:56,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:56,569 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,570 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/2d298e1625d5480f9d2e5120f150382e 2023-03-23 19:55:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:56,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:56,573 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:56,573 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:56,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/2f8d9d5f16ff40909d9d4018cb4d1f94 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94 2023-03-23 19:55:56,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94, entries=250, sequenceid=362, filesize=16.7 K 2023-03-23 19:55:56,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/33184b7de1e14dcba204a2f2228950b1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1 2023-03-23 19:55:56,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1, entries=150, sequenceid=362, filesize=11.9 K 2023-03-23 19:55:56,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/2d298e1625d5480f9d2e5120f150382e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e 2023-03-23 19:55:56,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e, entries=150, sequenceid=362, filesize=11.9 K 2023-03-23 19:55:56,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6b6146c7b551e398a832f6da27879933 in 855ms, sequenceid=362, compaction requested=true 2023-03-23 19:55:56,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:56,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:56,585 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:56,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:56,585 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:56,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:56,586 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42290 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:56,586 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:56,586 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:56,586 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:56,587 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,587 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5ac20451ce814fa8955463d1cb0a34e2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=36.5 K 2023-03-23 19:55:56,586 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,587 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/3fda712070954bc190cb1094159ce903, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=41.3 K 2023-03-23 19:55:56,587 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5ac20451ce814fa8955463d1cb0a34e2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:56,587 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 3fda712070954bc190cb1094159ce903, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:56,587 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 80b4d9509edc42d49a2c5cfc353239ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679601353575 2023-03-23 19:55:56,587 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4d07ec5100b94fee90b66df345b0c138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679601353575 2023-03-23 19:55:56,588 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 33184b7de1e14dcba204a2f2228950b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355726 2023-03-23 19:55:56,588 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 2f8d9d5f16ff40909d9d4018cb4d1f94, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355723 2023-03-23 19:55:56,601 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:56,602 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:56,618 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/17341267529842159e0bcf966013d53c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/17341267529842159e0bcf966013d53c 2023-03-23 19:55:56,622 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 17341267529842159e0bcf966013d53c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:56,622 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:56,622 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601356585; duration=0sec 2023-03-23 19:55:56,622 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:56,622 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:56,623 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:56,623 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:56,623 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,623 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/95553993607240d1beab6636ea54a018, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=36.5 K 2023-03-23 19:55:56,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 95553993607240d1beab6636ea54a018, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679601352934 2023-03-23 19:55:56,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 28c7adf95455490083a2c31b21a8ee93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679601353575 2023-03-23 19:55:56,624 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 2d298e1625d5480f9d2e5120f150382e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355726 2023-03-23 19:55:56,630 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:56,637 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/29492cd162ea46ae94c7311fde366ad3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/29492cd162ea46ae94c7311fde366ad3 2023-03-23 19:55:56,641 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 29492cd162ea46ae94c7311fde366ad3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:56,641 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:56,641 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601356585; duration=0sec 2023-03-23 19:55:56,641 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:56,673 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:56,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:56,677 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:56,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:56,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:56,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,677 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,678 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:56,678 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:56,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,679 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:56,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:56,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:56,680 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:56,680 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:55:56,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:56,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:56,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601416924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601416930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:56,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601416931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601417005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,018 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/6bf5e70fc4094ed499e652cdcf6cec96 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6bf5e70fc4094ed499e652cdcf6cec96 2023-03-23 19:55:57,021 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into 6bf5e70fc4094ed499e652cdcf6cec96(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:57,021 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:57,021 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601356585; duration=0sec 2023-03-23 19:55:57,021 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:57,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601417032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601417037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601417037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,093 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/fb216f960a344b2985a0bda223fc9eb0 2023-03-23 19:55:57,102 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/54eb8e1d428a434d84ef822ef0835a3a 2023-03-23 19:55:57,111 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/9178bfe937fd4d038a27b295e0362458 2023-03-23 19:55:57,114 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/fb216f960a344b2985a0bda223fc9eb0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0 2023-03-23 19:55:57,117 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0, entries=150, sequenceid=400, filesize=11.9 K 2023-03-23 19:55:57,118 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/54eb8e1d428a434d84ef822ef0835a3a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a 2023-03-23 19:55:57,121 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a, entries=150, sequenceid=400, filesize=11.9 K 2023-03-23 19:55:57,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/9178bfe937fd4d038a27b295e0362458 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458 2023-03-23 19:55:57,124 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458, entries=150, sequenceid=400, filesize=11.9 K 2023-03-23 19:55:57,125 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 6b6146c7b551e398a832f6da27879933 in 445ms, sequenceid=400, compaction requested=false 2023-03-23 19:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:57,125 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,125 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b68e4d9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,128 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,129 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,131 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,131 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,133 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,135 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,135 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,138 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,138 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,140 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,140 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,140 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,140 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,141 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,140 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,140 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,141 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:57,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:57,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:55:57,241 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:57,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:57,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,245 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,250 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,252 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,252 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,252 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,252 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abb525a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,257 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,257 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,258 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,265 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,266 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,268 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,269 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,270 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,275 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,275 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,275 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,275 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601417273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601417274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601417278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,375 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,379 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,380 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,381 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,381 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601417379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601417380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,383 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,384 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,384 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,384 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,386 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e438eb2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601417384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,390 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,393 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,393 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,394 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,395 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,395 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,396 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,399 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,399 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:57,400 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,400 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,400 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,400 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,400 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,500 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,503 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,503 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,504 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,504 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,504 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,505 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,509 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,510 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,511 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5986edb4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,513 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,513 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,517 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,521 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,522 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,522 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,522 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,522 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,523 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,523 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,523 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,524 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,525 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,527 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,527 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,527 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601417583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601417584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601417589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,627 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,630 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,630 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,631 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,632 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,632 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,634 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55449471[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,638 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,639 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,639 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,641 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,641 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,641 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,641 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,643 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,643 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,643 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/9ea17e48a745484685547efb30140150 2023-03-23 19:55:57,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,649 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,650 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,653 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,654 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,654 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,654 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:57,754 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,757 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,757 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,758 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,759 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,761 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,761 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,761 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,762 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,763 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556d3501[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,765 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,765 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,768 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,769 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,769 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,772 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,772 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,772 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,773 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,773 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,773 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,774 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,775 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,777 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,777 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,777 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:57,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,777 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,778 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,778 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,778 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,878 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:57,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:57,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:57,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:57,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:57,881 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:57,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:57,881 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:57,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,882 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,884 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,885 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:57,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:57,886 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,886 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:57,886 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:57,886 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:57,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:57,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:57,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,888 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7891700[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:57,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:57,890 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,891 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:57,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:57,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:57,892 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:57,893 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:57,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,893 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35840 deadline: 1679601417892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35782 deadline: 1679601417892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:57,897 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:57,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:57,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:57,897 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:57,897 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:57,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:57,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,899 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,900 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35766 deadline: 1679601417895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,901 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,906 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:57,906 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:57,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:57,907 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:57,907 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:57,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,907 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:57,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:57,907 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:57,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35756 deadline: 1679601417963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:57,971 DEBUG [Thread-2721] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:55:58,007 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:58,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:58,010 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:58,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:58,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:58,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:58,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,012 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:58,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:58,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:58,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:58,014 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:58,014 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:58,016 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,016 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:58,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@698ca108[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:58,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:58,018 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,020 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:58,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,020 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:55:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35812 deadline: 1679601418018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,021 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,021 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:58,024 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:58,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:58,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:58,024 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:58,024 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:58,025 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,025 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,025 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:58,026 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,028 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:58,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:58,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:58,034 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:58,034 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,034 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,034 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:58,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/5b396981cd39496fa1ce7dc9817bf73d 2023-03-23 19:55:58,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/ccd73578a6ee446c968e7a0d05d799a9 2023-03-23 19:55:58,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/9ea17e48a745484685547efb30140150 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150 2023-03-23 19:55:58,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150, entries=200, sequenceid=413, filesize=14.3 K 2023-03-23 19:55:58,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/5b396981cd39496fa1ce7dc9817bf73d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d 2023-03-23 19:55:58,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d, entries=150, sequenceid=413, filesize=11.9 K 2023-03-23 19:55:58,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/ccd73578a6ee446c968e7a0d05d799a9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9 2023-03-23 19:55:58,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9, entries=150, sequenceid=413, filesize=11.9 K 2023-03-23 19:55:58,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6b6146c7b551e398a832f6da27879933 in 839ms, sequenceid=413, compaction requested=true 2023-03-23 19:55:58,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:58,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:58,080 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:58,080 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:58,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:58,081 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:58,081 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:58,081 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:58,081 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/A is initiating minor compaction (all files) 2023-03-23 19:55:58,081 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/B is initiating minor compaction (all files) 2023-03-23 19:55:58,081 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/A in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,081 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/B in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,081 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/17341267529842159e0bcf966013d53c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=39.0 K 2023-03-23 19:55:58,081 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6bf5e70fc4094ed499e652cdcf6cec96, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=36.6 K 2023-03-23 19:55:58,082 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 17341267529842159e0bcf966013d53c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355726 2023-03-23 19:55:58,083 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 6bf5e70fc4094ed499e652cdcf6cec96, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355726 2023-03-23 19:55:58,083 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting fb216f960a344b2985a0bda223fc9eb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679601355823 2023-03-23 19:55:58,084 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 54eb8e1d428a434d84ef822ef0835a3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679601355823 2023-03-23 19:55:58,084 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9ea17e48a745484685547efb30140150, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679601356921 2023-03-23 19:55:58,084 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5b396981cd39496fa1ce7dc9817bf73d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679601356921 2023-03-23 19:55:58,089 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#A#compaction#459 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:58,090 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#B#compaction#460 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:58,104 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/31b39504ae2445a3a391b681fc39d96c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/31b39504ae2445a3a391b681fc39d96c 2023-03-23 19:55:58,107 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/B of 6b6146c7b551e398a832f6da27879933 into 31b39504ae2445a3a391b681fc39d96c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:58,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:58,108 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/B, priority=13, startTime=1679601358080; duration=0sec 2023-03-23 19:55:58,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:55:58,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:55:58,108 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:55:58,109 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 6b6146c7b551e398a832f6da27879933/C is initiating minor compaction (all files) 2023-03-23 19:55:58,109 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b6146c7b551e398a832f6da27879933/C in TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,109 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/29492cd162ea46ae94c7311fde366ad3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp, totalSize=36.6 K 2023-03-23 19:55:58,110 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 29492cd162ea46ae94c7311fde366ad3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679601355726 2023-03-23 19:55:58,110 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9178bfe937fd4d038a27b295e0362458, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679601355823 2023-03-23 19:55:58,111 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting ccd73578a6ee446c968e7a0d05d799a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679601356921 2023-03-23 19:55:58,116 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b6146c7b551e398a832f6da27879933#C#compaction#461 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:55:58,134 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:58,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:55:58,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:55:58,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:55:58,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:55:58,137 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:55:58,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:58,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:55:58,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:55:58,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,138 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:58,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:58,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:55:58,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:55:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:55:58,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:55:58,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:55:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:55:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:55:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:55:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:55:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:55:58,140 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,140 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. started... 2023-03-23 19:55:58,140 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:55:58,141 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:55:58,148 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/37828c220a89478cad71ac70705f5b3e 2023-03-23 19:55:58,248 DEBUG [Thread-2728] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:60301 2023-03-23 19:55:58,248 DEBUG [Thread-2736] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ac98310 to 127.0.0.1:60301 2023-03-23 19:55:58,248 DEBUG [Thread-2728] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,248 DEBUG [Thread-2734] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:60301 2023-03-23 19:55:58,248 DEBUG [Thread-2734] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,248 DEBUG [Thread-2736] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,251 DEBUG [Thread-2730] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:60301 2023-03-23 19:55:58,251 DEBUG [Thread-2730] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,252 DEBUG [Thread-2732] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:60301 2023-03-23 19:55:58,252 DEBUG [Thread-2732] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 6b6146c7b551e398a832f6da27879933 2023-03-23 19:55:58,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. as already flushing 2023-03-23 19:55:58,398 DEBUG [Thread-2725] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:60301 2023-03-23 19:55:58,398 DEBUG [Thread-2725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,400 DEBUG [Thread-2723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:60301 2023-03-23 19:55:58,400 DEBUG [Thread-2723] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,404 DEBUG [Thread-2717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:60301 2023-03-23 19:55:58,404 DEBUG [Thread-2717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:55:58,498 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/6b29c4f76806489f9b873d398045384c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6b29c4f76806489f9b873d398045384c 2023-03-23 19:55:58,501 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/A of 6b6146c7b551e398a832f6da27879933 into 6b29c4f76806489f9b873d398045384c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:58,501 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:58,501 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/A, priority=13, startTime=1679601358080; duration=0sec 2023-03-23 19:55:58,501 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:58,523 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/6cb3940f24d8417d9b659cb5ec323000 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/6cb3940f24d8417d9b659cb5ec323000 2023-03-23 19:55:58,526 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b6146c7b551e398a832f6da27879933/C of 6b6146c7b551e398a832f6da27879933 into 6cb3940f24d8417d9b659cb5ec323000(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:55:58,526 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:58,526 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933., storeName=6b6146c7b551e398a832f6da27879933/C, priority=13, startTime=1679601358081; duration=0sec 2023-03-23 19:55:58,526 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:55:58,556 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/779bf08c75a14655bd19fd5dbb0e9a37 2023-03-23 19:55:58,966 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c4b5046c44264a2fabcc4ce249890590 2023-03-23 19:55:58,969 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/37828c220a89478cad71ac70705f5b3e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/37828c220a89478cad71ac70705f5b3e 2023-03-23 19:55:58,971 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/37828c220a89478cad71ac70705f5b3e, entries=150, sequenceid=434, filesize=11.9 K 2023-03-23 19:55:58,972 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/779bf08c75a14655bd19fd5dbb0e9a37 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/779bf08c75a14655bd19fd5dbb0e9a37 2023-03-23 19:55:58,974 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/779bf08c75a14655bd19fd5dbb0e9a37, entries=150, sequenceid=434, filesize=11.9 K 2023-03-23 19:55:58,975 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/c4b5046c44264a2fabcc4ce249890590 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c4b5046c44264a2fabcc4ce249890590 2023-03-23 19:55:58,978 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c4b5046c44264a2fabcc4ce249890590, entries=150, sequenceid=434, filesize=11.9 K 2023-03-23 19:55:58,978 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for 6b6146c7b551e398a832f6da27879933 in 838ms, sequenceid=434, compaction requested=false 2023-03-23 19:55:58,978 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:55:58,978 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:55:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:55:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:55:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:55:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:55:58,980 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:55:58,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181a5754[Count = 0] remaining members to acquire global barrier 2023-03-23 19:55:58,981 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:55:58,981 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,983 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:55:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:55:58,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:55:58,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:55:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:55:58,985 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:55:58,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:55:58,986 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:55:58,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:55:58,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:55:58,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:55:58,986 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:55:58,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:55:58,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:58,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:55:58,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:58,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:55:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:55:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:55:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:55:58,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:55:58,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:55:58,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:55:58,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:55:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:55:58,992 DEBUG [Thread-2727] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:55:58,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,992 DEBUG [Thread-2727] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:55:58,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:55:58,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:55:59,092 DEBUG [Thread-2727] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:55:59,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:00,039 DEBUG [Thread-2719] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:60301 2023-03-23 19:56:00,039 DEBUG [Thread-2719] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:07,990 DEBUG [Thread-2721] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:60301 2023-03-23 19:56:07,990 DEBUG [Thread-2721] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:07,990 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2252 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6754 rows 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2275 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6817 rows 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2261 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6781 rows 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2245 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6733 rows 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(402): scanned 2249 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(403): verified 6747 rows 2023-03-23 19:56:07,991 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:56:07,991 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:60301 2023-03-23 19:56:07,991 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:07,993 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:56:07,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:56:07,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:07,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:07,997 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601367997"}]},"ts":"1679601367997"} 2023-03-23 19:56:07,998 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:56:08,000 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:56:08,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, UNASSIGN}] 2023-03-23 19:56:08,001 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, UNASSIGN 2023-03-23 19:56:08,002 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6b6146c7b551e398a832f6da27879933, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:08,002 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601368002"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601368002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601368002"}]},"ts":"1679601368002"} 2023-03-23 19:56:08,003 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:56:08,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:08,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b6146c7b551e398a832f6da27879933 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6b6146c7b551e398a832f6da27879933, disabling compactions & flushes 2023-03-23 19:56:08,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. after waiting 0 ms 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:56:08,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6b6146c7b551e398a832f6da27879933 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=A 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=B 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b6146c7b551e398a832f6da27879933, store=C 2023-03-23 19:56:08,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:08,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:08,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/90fd71d44b9d49a898940e90e53196d1 2023-03-23 19:56:08,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:08,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/ace36ef226314ffbb1cb51d9bad1b49d 2023-03-23 19:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:09,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/5189e5919c55477fb05d105e89bd8453 2023-03-23 19:56:09,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/A/90fd71d44b9d49a898940e90e53196d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/90fd71d44b9d49a898940e90e53196d1 2023-03-23 19:56:09,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/90fd71d44b9d49a898940e90e53196d1, entries=150, sequenceid=444, filesize=11.9 K 2023-03-23 19:56:09,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/B/ace36ef226314ffbb1cb51d9bad1b49d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ace36ef226314ffbb1cb51d9bad1b49d 2023-03-23 19:56:09,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ace36ef226314ffbb1cb51d9bad1b49d, entries=150, sequenceid=444, filesize=11.9 K 2023-03-23 19:56:09,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/.tmp/C/5189e5919c55477fb05d105e89bd8453 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5189e5919c55477fb05d105e89bd8453 2023-03-23 19:56:09,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5189e5919c55477fb05d105e89bd8453, entries=150, sequenceid=444, filesize=11.9 K 2023-03-23 19:56:09,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 6b6146c7b551e398a832f6da27879933 in 1237ms, sequenceid=444, compaction requested=true 2023-03-23 19:56:09,416 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e888262019554526a2c9ca256a621077, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/1dc4e66e41c948d8bd2437eff8949ca6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6f6fe2ef22034ed3bdbd337d6948d94e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/0d35ae88fd6f4bd19b1c1801fadf2e1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/f9e1e1f8265d46ff96b9ab720ed11b99, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/db00e784d5194dcbbd882e2328e5e147, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/3fda712070954bc190cb1094159ce903, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/17341267529842159e0bcf966013d53c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150] to archive 2023-03-23 19:56:09,417 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:09,418 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/8c9734a094834c9bb84877d137fd2148 2023-03-23 19:56:09,419 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e247d9b60ec04f59bdb846d3da3bd6fb 2023-03-23 19:56:09,420 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/a54e8c7683a843fc8e0a173979eac4d1 2023-03-23 19:56:09,421 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e888262019554526a2c9ca256a621077 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/e888262019554526a2c9ca256a621077 2023-03-23 19:56:09,421 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4066b25c5633427194abd110fd87ef4b 2023-03-23 19:56:09,422 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/ffe79afef80e441e9dd8100a5067906f 2023-03-23 19:56:09,423 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/1dc4e66e41c948d8bd2437eff8949ca6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/1dc4e66e41c948d8bd2437eff8949ca6 2023-03-23 19:56:09,423 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b1ddb2fcf2d24d24a6d25b56765e1a8b 2023-03-23 19:56:09,424 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/53170059e08c49eaacf8c76f00e014df 2023-03-23 19:56:09,425 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6f6fe2ef22034ed3bdbd337d6948d94e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6f6fe2ef22034ed3bdbd337d6948d94e 2023-03-23 19:56:09,425 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/00ae6a674f9845aba454aa43290b4af0 2023-03-23 19:56:09,426 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/b38e4d9f97f94e6288ad91fa97df8705 2023-03-23 19:56:09,427 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/42e01d3defcc4f9890ea6fd809b4b20b 2023-03-23 19:56:09,427 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/0d35ae88fd6f4bd19b1c1801fadf2e1f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/0d35ae88fd6f4bd19b1c1801fadf2e1f 2023-03-23 19:56:09,428 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6ae52678483a406c9586f749ab801d58 2023-03-23 19:56:09,429 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6664250b9f6a44e289c8d1adeb190797 2023-03-23 19:56:09,429 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/f9e1e1f8265d46ff96b9ab720ed11b99 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/f9e1e1f8265d46ff96b9ab720ed11b99 2023-03-23 19:56:09,430 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/5ada4ba061944859a243310838d7f987 2023-03-23 19:56:09,431 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c778b143d29b4650b3d2cd41bfd6b94c 2023-03-23 19:56:09,432 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/d5ca18753d804afbaf21db88af5a2344 2023-03-23 19:56:09,432 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/db00e784d5194dcbbd882e2328e5e147 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/db00e784d5194dcbbd882e2328e5e147 2023-03-23 19:56:09,433 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/bf8c1c79f027446a8a5c18deb352b0d9 2023-03-23 19:56:09,434 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/c1ac95630c5f427dbcda631208fa2d16 2023-03-23 19:56:09,434 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/7201b42667474e4384838b6ce9db9a45 2023-03-23 19:56:09,435 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/3fda712070954bc190cb1094159ce903 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/3fda712070954bc190cb1094159ce903 2023-03-23 19:56:09,436 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/75cd8368d6d14c4c9caacad3da5511f9 2023-03-23 19:56:09,437 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/4d07ec5100b94fee90b66df345b0c138 2023-03-23 19:56:09,438 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/2f8d9d5f16ff40909d9d4018cb4d1f94 2023-03-23 19:56:09,439 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/17341267529842159e0bcf966013d53c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/17341267529842159e0bcf966013d53c 2023-03-23 19:56:09,440 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/fb216f960a344b2985a0bda223fc9eb0 2023-03-23 19:56:09,440 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/9ea17e48a745484685547efb30140150 2023-03-23 19:56:09,443 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e6da374ffcc6408896b85f5ab2cb3155, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c8ee0a7d4ba1488892038b43e0109a77, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c949a149cb5b4d17b30528dc2dc44da7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/abd25c3077c149ed955fdc74a133d82c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3d3ea6a34b9d47e2996002251e1598c7, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5ac20451ce814fa8955463d1cb0a34e2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6bf5e70fc4094ed499e652cdcf6cec96, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d] to archive 2023-03-23 19:56:09,444 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:09,445 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/af73f36fd1f9406aa3acd2541fc1ec01 2023-03-23 19:56:09,446 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/a3eecf50fac84bfab9202ef00f8c43eb 2023-03-23 19:56:09,447 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e6da374ffcc6408896b85f5ab2cb3155 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e6da374ffcc6408896b85f5ab2cb3155 2023-03-23 19:56:09,448 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/004c8966145649f1894522f4d7ec5abd 2023-03-23 19:56:09,449 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3a92188616474fa38a5f13eb80ebd8dc 2023-03-23 19:56:09,450 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c8ee0a7d4ba1488892038b43e0109a77 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c8ee0a7d4ba1488892038b43e0109a77 2023-03-23 19:56:09,451 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/d9f16eb524e347d4959bddfb45ce2f54 2023-03-23 19:56:09,451 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/fb7ed1900513437eb3c0444181478361 2023-03-23 19:56:09,452 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/b2b62ebf6eda4ec2a8b01ab38b9ca6fa 2023-03-23 19:56:09,453 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8b32bcc897d347aaac563ffa96ea6912 2023-03-23 19:56:09,453 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/071e6e06fb174dd58b7578b41773bc71 2023-03-23 19:56:09,454 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6c6b64e724df40f2a496eb8429cec3d1 2023-03-23 19:56:09,455 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c949a149cb5b4d17b30528dc2dc44da7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c949a149cb5b4d17b30528dc2dc44da7 2023-03-23 19:56:09,456 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/cb36edfeeaa04c5d9d46d76f3142d3cd 2023-03-23 19:56:09,456 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/55247cecede3442391e0af2c89a10ffa 2023-03-23 19:56:09,457 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/abd25c3077c149ed955fdc74a133d82c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/abd25c3077c149ed955fdc74a133d82c 2023-03-23 19:56:09,458 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/8acedb5cb4344855a30b34dc9e6499be 2023-03-23 19:56:09,458 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/bd3014aff6a04ee48a6d6f623610e5a9 2023-03-23 19:56:09,459 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/133f10aec2304127aa68eeeee26960b1 2023-03-23 19:56:09,460 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3d3ea6a34b9d47e2996002251e1598c7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/3d3ea6a34b9d47e2996002251e1598c7 2023-03-23 19:56:09,461 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/c509d5035a294de5972d79c05eeadfa2 2023-03-23 19:56:09,461 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ccd272f193a04119b8890a193fb92a15 2023-03-23 19:56:09,462 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/e84c6cccd331433bb1eea36a03006854 2023-03-23 19:56:09,463 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/610c7bc0c9024f34b5d510baff26c60e 2023-03-23 19:56:09,464 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5ac20451ce814fa8955463d1cb0a34e2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5ac20451ce814fa8955463d1cb0a34e2 2023-03-23 19:56:09,464 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/dd63fd4b686a467b82e36c39d55f3d2f 2023-03-23 19:56:09,465 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/80b4d9509edc42d49a2c5cfc353239ad 2023-03-23 19:56:09,466 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6bf5e70fc4094ed499e652cdcf6cec96 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/6bf5e70fc4094ed499e652cdcf6cec96 2023-03-23 19:56:09,467 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/33184b7de1e14dcba204a2f2228950b1 2023-03-23 19:56:09,467 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/54eb8e1d428a434d84ef822ef0835a3a 2023-03-23 19:56:09,468 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/5b396981cd39496fa1ce7dc9817bf73d 2023-03-23 19:56:09,471 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28be63099956421a885bb38d8417315e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cb5d240cb4234385ae8dfe82fe711e03, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/217af493e7114578b3e3775d771129b9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f50065ebc17c46ed8b997ea58c81d4bb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/eb1018997afa4ab7a9c261f723f61702, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/50c9f948e94a46f9923e45da0ec911ff, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/95553993607240d1beab6636ea54a018, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/29492cd162ea46ae94c7311fde366ad3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9] to archive 2023-03-23 19:56:09,471 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:09,473 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/689bedacd0334aff84cfb3f65a5ce14d 2023-03-23 19:56:09,473 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/7460afdfd49749c19a33e69eec0f0b07 2023-03-23 19:56:09,474 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28be63099956421a885bb38d8417315e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28be63099956421a885bb38d8417315e 2023-03-23 19:56:09,475 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5dbebe92901e4a9bba74f7da22538064 2023-03-23 19:56:09,476 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/588e065066fb46f8903fd34c6c216027 2023-03-23 19:56:09,477 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cb5d240cb4234385ae8dfe82fe711e03 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cb5d240cb4234385ae8dfe82fe711e03 2023-03-23 19:56:09,477 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c62da57c88f04d20a17fe22a95ab1344 2023-03-23 19:56:09,478 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/a272081c20b748048740aaf22ad266ca 2023-03-23 19:56:09,479 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/217af493e7114578b3e3775d771129b9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/217af493e7114578b3e3775d771129b9 2023-03-23 19:56:09,480 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/735e192b5f474d01bb3aab4dc9b287ca 2023-03-23 19:56:09,481 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f11ec8d5131d4cceba7026a6371f13e4 2023-03-23 19:56:09,481 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/cfb5404bc5454f7293c105214c1011ca 2023-03-23 19:56:09,482 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f50065ebc17c46ed8b997ea58c81d4bb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f50065ebc17c46ed8b997ea58c81d4bb 2023-03-23 19:56:09,483 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c9fed8839c3e4486a01669f024ffd18a 2023-03-23 19:56:09,484 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/bc58a819d463451394b8d2d73ef9c6dc 2023-03-23 19:56:09,484 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/eb1018997afa4ab7a9c261f723f61702 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/eb1018997afa4ab7a9c261f723f61702 2023-03-23 19:56:09,485 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/35581c85a4fe4ee6a81c7863d1ec130f 2023-03-23 19:56:09,486 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/84bed03268c045c589fb1d06de031097 2023-03-23 19:56:09,486 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/f31fcdea2d7d41128aebbf505bd3f294 2023-03-23 19:56:09,487 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/50c9f948e94a46f9923e45da0ec911ff to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/50c9f948e94a46f9923e45da0ec911ff 2023-03-23 19:56:09,488 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/adda84041dfe4d32a455ea90207cd1a0 2023-03-23 19:56:09,489 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/49d480d08ce14c8cb3b3452c0d0a4916 2023-03-23 19:56:09,490 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/3e4bbbca492b4a25b2469aed1bac0621 2023-03-23 19:56:09,490 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/b5a9d4971313464885a126d3bb7ef9c3 2023-03-23 19:56:09,491 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/95553993607240d1beab6636ea54a018 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/95553993607240d1beab6636ea54a018 2023-03-23 19:56:09,492 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/da3763ad3e254d42965610b16707436c 2023-03-23 19:56:09,493 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/28c7adf95455490083a2c31b21a8ee93 2023-03-23 19:56:09,493 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/29492cd162ea46ae94c7311fde366ad3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/29492cd162ea46ae94c7311fde366ad3 2023-03-23 19:56:09,494 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/2d298e1625d5480f9d2e5120f150382e 2023-03-23 19:56:09,495 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/9178bfe937fd4d038a27b295e0362458 2023-03-23 19:56:09,496 DEBUG [StoreCloser-TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/ccd73578a6ee446c968e7a0d05d799a9 2023-03-23 19:56:09,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/recovered.edits/447.seqid, newMaxSeqId=447, maxSeqId=1 2023-03-23 19:56:09,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933. 2023-03-23 19:56:09,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6b6146c7b551e398a832f6da27879933: 2023-03-23 19:56:09,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b6146c7b551e398a832f6da27879933 2023-03-23 19:56:09,502 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6b6146c7b551e398a832f6da27879933, regionState=CLOSED 2023-03-23 19:56:09,503 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601369502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601369502"}]},"ts":"1679601369502"} 2023-03-23 19:56:09,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-23 19:56:09,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 6b6146c7b551e398a832f6da27879933, server=jenkins-hbase4.apache.org,34277,1679601218015 in 1.5000 sec 2023-03-23 19:56:09,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-23 19:56:09,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b6146c7b551e398a832f6da27879933, UNASSIGN in 1.5040 sec 2023-03-23 19:56:09,506 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601369506"}]},"ts":"1679601369506"} 2023-03-23 19:56:09,507 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:56:09,508 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:56:09,509 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5160 sec 2023-03-23 19:56:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-23 19:56:10,101 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-23 19:56:10,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:56:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,104 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-23 19:56:10,104 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,109 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:56:10,111 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/recovered.edits] 2023-03-23 19:56:10,114 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/37828c220a89478cad71ac70705f5b3e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/37828c220a89478cad71ac70705f5b3e 2023-03-23 19:56:10,115 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6b29c4f76806489f9b873d398045384c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/6b29c4f76806489f9b873d398045384c 2023-03-23 19:56:10,116 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/90fd71d44b9d49a898940e90e53196d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/A/90fd71d44b9d49a898940e90e53196d1 2023-03-23 19:56:10,117 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/31b39504ae2445a3a391b681fc39d96c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/31b39504ae2445a3a391b681fc39d96c 2023-03-23 19:56:10,118 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/779bf08c75a14655bd19fd5dbb0e9a37 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/779bf08c75a14655bd19fd5dbb0e9a37 2023-03-23 19:56:10,119 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ace36ef226314ffbb1cb51d9bad1b49d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/B/ace36ef226314ffbb1cb51d9bad1b49d 2023-03-23 19:56:10,121 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5189e5919c55477fb05d105e89bd8453 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/5189e5919c55477fb05d105e89bd8453 2023-03-23 19:56:10,122 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/6cb3940f24d8417d9b659cb5ec323000 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/6cb3940f24d8417d9b659cb5ec323000 2023-03-23 19:56:10,123 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c4b5046c44264a2fabcc4ce249890590 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/C/c4b5046c44264a2fabcc4ce249890590 2023-03-23 19:56:10,125 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/recovered.edits/447.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933/recovered.edits/447.seqid 2023-03-23 19:56:10,126 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/6b6146c7b551e398a832f6da27879933 2023-03-23 19:56:10,126 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:56:10,128 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,129 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:56:10,130 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:56:10,131 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,131 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:56:10,131 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601370131"}]},"ts":"9223372036854775807"} 2023-03-23 19:56:10,133 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:56:10,133 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6b6146c7b551e398a832f6da27879933, NAME => 'TestAcidGuarantees,,1679601336083.6b6146c7b551e398a832f6da27879933.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:56:10,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:56:10,133 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601370133"}]},"ts":"9223372036854775807"} 2023-03-23 19:56:10,134 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:56:10,135 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,136 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-03-23 19:56:10,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-23 19:56:10,205 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-23 19:56:10,216 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 224) - Thread LEAK? -, OpenFileDescriptor=518 (was 502) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 294), ProcessCount=169 (was 171), AvailableMemoryMB=5503 (was 5370) - AvailableMemoryMB LEAK? - 2023-03-23 19:56:10,227 INFO [Listener at localhost/45187] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=518, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=169, AvailableMemoryMB=5503 2023-03-23 19:56:10,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:56:10,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:56:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:10,232 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-23 19:56:10,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-23 19:56:10,232 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-23 19:56:10,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:10,233 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:10,234 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 empty. 2023-03-23 19:56:10,234 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:10,234 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:56:10,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:10,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:10,641 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:56:10,643 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 92074693d0baf272335970fad69a35c6, NAME => 'TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp 2023-03-23 19:56:10,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 92074693d0baf272335970fad69a35c6, disabling compactions & flushes 2023-03-23 19:56:11,048 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. after waiting 0 ms 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,048 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,048 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:11,050 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-23 19:56:11,050 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601371050"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601371050"}]},"ts":"1679601371050"} 2023-03-23 19:56:11,051 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-23 19:56:11,052 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-23 19:56:11,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601371052"}]},"ts":"1679601371052"} 2023-03-23 19:56:11,053 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-23 19:56:11,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, ASSIGN}] 2023-03-23 19:56:11,057 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, ASSIGN 2023-03-23 19:56:11,058 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=false 2023-03-23 19:56:11,209 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:11,209 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601371209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601371209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601371209"}]},"ts":"1679601371209"} 2023-03-23 19:56:11,210 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:56:11,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:11,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92074693d0baf272335970fad69a35c6, NAME => 'TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:56:11,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:56:11,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,366 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,367 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A 2023-03-23 19:56:11,367 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A 2023-03-23 19:56:11,367 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:11,367 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName A 2023-03-23 19:56:11,367 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:11,368 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,368 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B 2023-03-23 19:56:11,368 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B 2023-03-23 19:56:11,369 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:11,369 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName B 2023-03-23 19:56:11,369 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:11,369 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,370 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C 2023-03-23 19:56:11,370 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C 2023-03-23 19:56:11,370 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:11,370 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName C 2023-03-23 19:56:11,371 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:11,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:56:11,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:11,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-23 19:56:11,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 92074693d0baf272335970fad69a35c6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67298843, jitterRate=0.002830907702445984}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:56:11,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:11,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., pid=56, masterSystemTime=1679601371361 2023-03-23 19:56:11,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:11,377 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:11,377 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601371377"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601371377"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601371377"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601371377"}]},"ts":"1679601371377"} 2023-03-23 19:56:11,379 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-23 19:56:11,379 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 in 168 msec 2023-03-23 19:56:11,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-23 19:56:11,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, ASSIGN in 323 msec 2023-03-23 19:56:11,381 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-23 19:56:11,381 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601371381"}]},"ts":"1679601371381"} 2023-03-23 19:56:11,382 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-23 19:56:11,383 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-23 19:56:11,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1530 sec 2023-03-23 19:56:12,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-23 19:56:12,336 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-23 19:56:12,337 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,342 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,343 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,344 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48250, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,345 DEBUG [Listener at localhost/45187] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-23 19:56:12,347 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38722, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-23 19:56:12,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-23 19:56:12,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-23 19:56:12,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:12,356 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:56:12,357 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-23 19:56:12,357 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-23 19:56:12,359 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-23 19:56:12,361 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, REOPEN/MOVE}] 2023-03-23 19:56:12,361 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, REOPEN/MOVE 2023-03-23 19:56:12,362 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,362 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601372362"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601372362"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601372362"}]},"ts":"1679601372362"} 2023-03-23 19:56:12,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:56:12,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 92074693d0baf272335970fad69a35c6, disabling compactions & flushes 2023-03-23 19:56:12,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. after waiting 0 ms 2023-03-23 19:56:12,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-23 19:56:12,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:12,519 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 92074693d0baf272335970fad69a35c6 to self. 2023-03-23 19:56:12,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,520 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=CLOSED 2023-03-23 19:56:12,520 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601372520"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601372520"}]},"ts":"1679601372520"} 2023-03-23 19:56:12,522 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-23 19:56:12,522 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 in 158 msec 2023-03-23 19:56:12,523 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34277,1679601218015; forceNewPlan=false, retain=true 2023-03-23 19:56:12,673 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,673 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601372673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601372673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601372673"}]},"ts":"1679601372673"} 2023-03-23 19:56:12,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:56:12,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92074693d0baf272335970fad69a35c6, NAME => 'TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.', STARTKEY => '', ENDKEY => ''} 2023-03-23 19:56:12,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-23 19:56:12,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,829 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,830 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A 2023-03-23 19:56:12,830 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A 2023-03-23 19:56:12,830 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:12,830 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName A 2023-03-23 19:56:12,831 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:12,832 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,832 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B 2023-03-23 19:56:12,832 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B 2023-03-23 19:56:12,832 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:12,833 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName B 2023-03-23 19:56:12,833 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:12,833 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,833 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C 2023-03-23 19:56:12,834 DEBUG [StoreOpener-92074693d0baf272335970fad69a35c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C 2023-03-23 19:56:12,834 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-23 19:56:12,834 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92074693d0baf272335970fad69a35c6 columnFamilyName C 2023-03-23 19:56:12,834 INFO [StoreOpener-92074693d0baf272335970fad69a35c6-1] regionserver.HStore(310): Store=92074693d0baf272335970fad69a35c6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-23 19:56:12,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-23 19:56:12,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 92074693d0baf272335970fad69a35c6; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68465691, jitterRate=0.020218297839164734}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-23 19:56:12,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:12,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., pid=61, masterSystemTime=1679601372825 2023-03-23 19:56:12,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,840 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,840 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601372840"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679601372840"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679601372840"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679601372840"}]},"ts":"1679601372840"} 2023-03-23 19:56:12,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-23 19:56:12,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 in 167 msec 2023-03-23 19:56:12,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-23 19:56:12,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, REOPEN/MOVE in 481 msec 2023-03-23 19:56:12,844 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-23 19:56:12,844 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 484 msec 2023-03-23 19:56:12,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 497 msec 2023-03-23 19:56:12,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-23 19:56:12,847 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,851 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,852 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,856 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,856 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,859 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,860 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,863 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,863 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,866 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,867 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,871 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,872 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,876 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,876 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,882 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,882 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,886 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,886 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(139): Connect 0x62d52321 to 127.0.0.1:60301 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-23 19:56:12,892 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eedd898, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-23 19:56:12,897 DEBUG [hconnection-0x5385965a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,897 DEBUG [hconnection-0x6fefa8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,897 DEBUG [hconnection-0x76366d07-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,897 DEBUG [hconnection-0x1c561f13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,901 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48260, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:12,903 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48262, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,903 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,903 DEBUG [hconnection-0x3849f5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,903 DEBUG [hconnection-0x4022fbd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,905 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,906 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,907 DEBUG [hconnection-0x5fd140c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,907 DEBUG [hconnection-0x9e624d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,907 DEBUG [hconnection-0x4b52607d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,907 DEBUG [hconnection-0x22031bae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-23 19:56:12,906 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48316, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:12,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:12,910 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:12,910 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:12,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48320, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,914 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48318, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48322, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,913 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:12,916 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:12,916 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48338, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-23 19:56:12,917 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:12,917 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:12,920 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:12,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:12,920 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:12,921 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,921 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:12,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:12,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:12,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:12,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:12,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:12,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:12,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:12,922 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:12,922 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:12,922 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:12,923 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-23 19:56:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:12,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:12,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:12,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601432937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601432937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601372932, max timestamp=1679601372939Num uniques 859; 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601372932, max timestamp=1679601372939Num uniques 859; 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679601372932, max timestamp=1679601372939Num uniques 859; 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:56:12,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-23 19:56:12,944 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303235d184dbbbe1f44a682e07b0d041aa5c7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235d184dbbbe1f44a682e07b0d041aa5c7 2023-03-23 19:56:12,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:12,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601432960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:12,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601432963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:12,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:12,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601432963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601433042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601433042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601433064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601433066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601433066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601433244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601433245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601433266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601433268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601433269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,350 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/871d42a26daf4debbe6e1ef2b5bf42f6 2023-03-23 19:56:13,364 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d01f6df6867e442d8552cb99ed8cbb3c 2023-03-23 19:56:13,373 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3cf78b69aa334e32abdef44f230a6caf 2023-03-23 19:56:13,377 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/871d42a26daf4debbe6e1ef2b5bf42f6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6 2023-03-23 19:56:13,380 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6, entries=150, sequenceid=15, filesize=25.1 K 2023-03-23 19:56:13,380 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d01f6df6867e442d8552cb99ed8cbb3c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c 2023-03-23 19:56:13,383 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c, entries=150, sequenceid=15, filesize=11.6 K 2023-03-23 19:56:13,384 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3cf78b69aa334e32abdef44f230a6caf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf 2023-03-23 19:56:13,387 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf, entries=150, sequenceid=15, filesize=11.6 K 2023-03-23 19:56:13,387 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92074693d0baf272335970fad69a35c6 in 464ms, sequenceid=15, compaction requested=false 2023-03-23 19:56:13,387 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-23 19:56:13,388 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:13,388 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:13,390 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24e3ac6b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:13,391 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:13,392 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,392 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:13,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:13,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,393 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:13,393 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:13,396 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:13,399 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:13,399 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:13,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:13,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:13,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:13,405 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,406 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,408 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:13,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:13,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:13,408 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:13,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:13,408 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:13,408 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:13,409 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:13,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:13,409 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:13,409 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,509 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:13,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:13,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:13,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:13,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:13,512 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:13,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:13,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:13,512 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:13,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,513 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:13,515 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,515 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:13,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:13,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:13,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:13,516 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:13,516 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:13,517 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:13,517 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:13,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:13,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:13,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601433558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601433561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601433570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601433571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601433572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601433662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601433665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601433866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:13,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601433869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:13,928 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303233c05a676a6c84e8fb9dc0c5068407032 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233c05a676a6c84e8fb9dc0c5068407032 2023-03-23 19:56:14,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601434074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601434075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601434079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601434168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601434173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,335 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/5ee736e9d0254f22b0d6070b0238374b 2023-03-23 19:56:14,344 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4fadf89f275f496194537176931016ad 2023-03-23 19:56:14,352 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/73f1910185854d3eb22c16b380b1497f 2023-03-23 19:56:14,355 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/5ee736e9d0254f22b0d6070b0238374b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b 2023-03-23 19:56:14,358 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b, entries=150, sequenceid=51, filesize=25.1 K 2023-03-23 19:56:14,359 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4fadf89f275f496194537176931016ad as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad 2023-03-23 19:56:14,362 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad, entries=150, sequenceid=51, filesize=11.6 K 2023-03-23 19:56:14,363 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/73f1910185854d3eb22c16b380b1497f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f 2023-03-23 19:56:14,366 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f, entries=150, sequenceid=51, filesize=11.6 K 2023-03-23 19:56:14,366 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 849ms, sequenceid=51, compaction requested=false 2023-03-23 19:56:14,366 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:14,366 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:14,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:14,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:14,366 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:14,367 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:14,368 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:14,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,368 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:14,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:14,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1d8ee0[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:14,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:14,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,371 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:14,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:14,372 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:14,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:14,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:14,375 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:14,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:14,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:14,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:14,375 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:14,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:14,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:14,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,379 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:14,381 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:14,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:14,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:14,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:14,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:14,382 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,382 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:14,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:14,482 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:14,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:14,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:14,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:14,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:14,486 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:14,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:14,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:14,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:14,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,486 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:14,487 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:14,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:14,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:14,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:14,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:14,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:14,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:14,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:14,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:14,489 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:14,489 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:14,490 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:14,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:14,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:14,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601374685, max timestamp=1679601374696Num uniques 859; 2023-03-23 19:56:14,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601374685, max timestamp=1679601374696Num uniques 859; 2023-03-23 19:56:14,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601374685, max timestamp=1679601374696Num uniques 859; 2023-03-23 19:56:14,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:14,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:14,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:14,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601434722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601434725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601434826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:14,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601434830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:14,901 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303233adb463655454fe8baf95c8a768de76b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233adb463655454fe8baf95c8a768de76b 2023-03-23 19:56:15,024 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 19:56:15,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601435029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601435034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601435080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601435082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601435088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,306 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/aa998607cea14b02b7e4ddd746f9cb4f 2023-03-23 19:56:15,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601435333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601435339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,716 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4788f02deddd431595bb483171c2e721 2023-03-23 19:56:15,725 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/9d5ca36aebc84b24bd5b2278e1d817df 2023-03-23 19:56:15,728 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/aa998607cea14b02b7e4ddd746f9cb4f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f 2023-03-23 19:56:15,731 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f, entries=150, sequenceid=62, filesize=25.1 K 2023-03-23 19:56:15,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4788f02deddd431595bb483171c2e721 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721 2023-03-23 19:56:15,734 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721, entries=150, sequenceid=62, filesize=11.6 K 2023-03-23 19:56:15,735 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/9d5ca36aebc84b24bd5b2278e1d817df as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df 2023-03-23 19:56:15,738 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df, entries=150, sequenceid=62, filesize=11.6 K 2023-03-23 19:56:15,739 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92074693d0baf272335970fad69a35c6 in 1250ms, sequenceid=62, compaction requested=true 2023-03-23 19:56:15,739 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:15,739 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:15,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:15,740 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,741 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:15,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6795f66a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:15,742 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:15,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,745 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:15,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:15,745 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:15,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,745 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:15,746 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:15,746 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:15,746 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,746 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:15,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:15,749 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:15,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:15,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:15,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:15,749 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:15,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:15,750 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:15,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,752 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:15,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:15,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:15,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:15,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,755 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:15,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:15,755 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:15,755 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:15,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:15,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:15,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:15,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:15,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:15,850 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323565d29009e1e4491abda6f517969f242 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323565d29009e1e4491abda6f517969f242 2023-03-23 19:56:15,855 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:15,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:15,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:15,858 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:15,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:15,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:15,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:15,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601435856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601435856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:15,860 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:15,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:15,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:15,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:15,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:15,862 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:15,862 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:15,862 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:15,862 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:15,862 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:15,863 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,864 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dffbd3b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:15,865 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:15,865 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,866 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:15,866 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:15,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,866 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:15,868 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:15,868 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:15,870 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:15,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:15,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:15,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:15,870 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:15,872 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,872 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,872 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:15,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,874 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:15,879 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:15,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:15,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:15,879 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,879 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:15,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,880 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601435960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:15,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601435960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,979 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:15,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:15,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:15,982 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:15,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:15,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:15,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:15,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,983 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,987 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:15,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:15,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,988 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:15,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:15,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:15,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:15,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:15,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:15,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:15,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:15,989 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:15,989 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,991 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,991 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1446cc1b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:15,992 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:15,992 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,993 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:15,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:15,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:15,995 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,995 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:15,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:15,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:15,997 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:15,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:15,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:15,997 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:15,997 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:15,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:15,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:15,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,001 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,003 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,003 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,003 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,103 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,106 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,108 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,110 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,111 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,113 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@132b2afd[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,113 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,115 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,115 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,116 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,119 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,119 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,121 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,122 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,124 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,125 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,125 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,125 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601436164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601436164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,225 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,228 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,228 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,228 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,228 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,229 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,231 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,233 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,235 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b45a854[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,237 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,237 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,238 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,238 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,240 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,242 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,242 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,242 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,246 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,248 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:16,249 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,249 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,256 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/494bcd2e9719488eb2e976f64989ca91 2023-03-23 19:56:16,349 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,352 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,352 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,353 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,354 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,355 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,356 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,356 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,359 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b83d062[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,361 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,362 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,362 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,362 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,362 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,364 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,366 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,366 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,366 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,371 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,377 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:16,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,378 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,378 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,378 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,378 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,378 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601436467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601436468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,478 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,481 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,481 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,482 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,482 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,483 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,483 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,484 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,484 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,484 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,484 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,484 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,486 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,486 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,488 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2987f31d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,489 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,492 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,492 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,494 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,494 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,494 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,496 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,497 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,498 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,500 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,500 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,500 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,600 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,604 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,604 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,605 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,605 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,606 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,606 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,610 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,610 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,611 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b15812[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,612 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,613 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,613 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,613 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,614 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,614 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,614 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,614 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,617 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,617 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,617 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,618 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,619 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,620 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,624 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,624 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,624 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,624 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/33ab07337fcf4e79b355a529019847dd 2023-03-23 19:56:16,725 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,728 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,728 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,729 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,731 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,731 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,735 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,735 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,736 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,736 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c04b38[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,737 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,739 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,739 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,739 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,740 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,742 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,743 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,745 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,745 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,745 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,746 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,747 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,749 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,749 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,749 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,750 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,850 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,853 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,853 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,855 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,857 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,857 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,857 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,857 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,857 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559922[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,861 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,863 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,863 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,863 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,865 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,867 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,867 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,869 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,873 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,873 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,973 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:16,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:16,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:16,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:16,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:16,976 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:16,976 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,976 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:16,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:16,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601436974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:16,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601436975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,979 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:16,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:16,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:16,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:16,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:16,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:16,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:16,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:16,983 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:16,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16dc9d2e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:16,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:16,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:16,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:16,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:16,987 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:16,989 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:16,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:16,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:16,989 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:16,989 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:16,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:16,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:16,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:16,996 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:16,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:16,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:16,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:16,996 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:16,996 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:16,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:16,996 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:16,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/904e30256a504b90afceee2a252de882 2023-03-23 19:56:17,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/494bcd2e9719488eb2e976f64989ca91 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91 2023-03-23 19:56:17,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91, entries=200, sequenceid=98, filesize=31.9 K 2023-03-23 19:56:17,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/33ab07337fcf4e79b355a529019847dd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd 2023-03-23 19:56:17,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd, entries=150, sequenceid=98, filesize=11.6 K 2023-03-23 19:56:17,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/904e30256a504b90afceee2a252de882 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882 2023-03-23 19:56:17,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882, entries=150, sequenceid=98, filesize=11.6 K 2023-03-23 19:56:17,089 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 1253ms, sequenceid=98, compaction requested=true 2023-03-23 19:56:17,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:17,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:17,089 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:17,089 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:17,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:17,090 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:17,096 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,097 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:17,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,097 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:17,097 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,097 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=46.4 K 2023-03-23 19:56:17,098 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:17,098 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:17,098 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,098 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=107.3 K 2023-03-23 19:56:17,098 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d01f6df6867e442d8552cb99ed8cbb3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601372920 2023-03-23 19:56:17,099 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 871d42a26daf4debbe6e1ef2b5bf42f6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601372920 2023-03-23 19:56:17,099 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4fadf89f275f496194537176931016ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601372958 2023-03-23 19:56:17,099 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5ee736e9d0254f22b0d6070b0238374b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601372958 2023-03-23 19:56:17,099 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting aa998607cea14b02b7e4ddd746f9cb4f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679601373553 2023-03-23 19:56:17,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,100 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 494bcd2e9719488eb2e976f64989ca91, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374717 2023-03-23 19:56:17,100 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4788f02deddd431595bb483171c2e721, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679601373553 2023-03-23 19:56:17,101 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 33ab07337fcf4e79b355a529019847dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374721 2023-03-23 19:56:17,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:17,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:17,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:17,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,107 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,107 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,109 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,129 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:17,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601437132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601437133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601437136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,164 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,165 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,166 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,166 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,172 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,173 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc4362f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,180 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,180 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,184 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,186 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#481 average throughput is 0.39 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,189 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,189 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,190 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,192 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,196 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,199 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:17,200 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,200 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,238 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/be09fc6cd0f7495489bf63a61e22abfc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/be09fc6cd0f7495489bf63a61e22abfc 2023-03-23 19:56:17,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601437237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601437238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,243 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into be09fc6cd0f7495489bf63a61e22abfc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:17,243 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:17,243 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=12, startTime=1679601377089; duration=0sec 2023-03-23 19:56:17,243 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:17,243 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601437241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,248 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:17,249 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:17,249 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,249 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=46.4 K 2023-03-23 19:56:17,249 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3cf78b69aa334e32abdef44f230a6caf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679601372920 2023-03-23 19:56:17,249 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 73f1910185854d3eb22c16b380b1497f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679601372958 2023-03-23 19:56:17,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9d5ca36aebc84b24bd5b2278e1d817df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679601373553 2023-03-23 19:56:17,250 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 904e30256a504b90afceee2a252de882, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374721 2023-03-23 19:56:17,261 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:17,275 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c7ee37bc14f74a53a62e165890e3d5bb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7ee37bc14f74a53a62e165890e3d5bb 2023-03-23 19:56:17,279 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into c7ee37bc14f74a53a62e165890e3d5bb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:17,279 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:17,279 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=12, startTime=1679601377090; duration=0sec 2023-03-23 19:56:17,280 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:17,300 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,304 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,304 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,305 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,307 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,308 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,320 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,320 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ca3b64[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,325 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,327 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,330 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,330 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,330 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,330 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,330 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,341 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,342 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,342 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,442 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601437441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,446 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,447 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601437444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601437445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,448 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fa23d6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,458 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,463 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,463 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,466 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,470 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,470 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,470 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,570 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,573 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,580 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,582 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,582 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,582 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,582 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,582 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bde38a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,587 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,592 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,592 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,592 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,597 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,599 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,601 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,602 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,602 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:17,602 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,602 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,609 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303239a2724b613ad4cf29c0ac7e9c38b83be to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239a2724b613ad4cf29c0ac7e9c38b83be 2023-03-23 19:56:17,677 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/09712bb24cb840b1af26625b62251cb2 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09712bb24cb840b1af26625b62251cb2 2023-03-23 19:56:17,688 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 09712bb24cb840b1af26625b62251cb2(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:17,688 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:17,688 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=12, startTime=1679601377089; duration=0sec 2023-03-23 19:56:17,688 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:17,702 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,705 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,706 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,724 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,724 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,724 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,724 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,727 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,727 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,727 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,727 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6816ce3c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,729 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,730 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,732 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,736 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,736 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,736 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,737 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,737 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,737 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,738 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,740 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,743 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:17,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,743 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,744 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,744 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,744 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601437746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601437749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601437750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,844 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,847 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,847 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,847 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,847 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,849 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,850 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,850 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,850 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,852 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,852 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,852 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,852 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,852 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,865 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,866 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,866 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513ec941[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,869 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,869 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:17,869 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:17,869 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:17,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,870 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,871 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,871 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:17,873 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:17,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:17,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:17,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:17,873 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:17,875 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,875 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,875 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,876 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,878 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:17,878 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:17,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:17,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,879 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:17,879 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:17,879 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:17,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:17,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601437978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,979 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:17,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:17,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:17,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601437979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:17,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:17,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:17,984 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:17,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:17,984 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:17,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:17,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:17,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,987 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:17,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:17,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:17,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:17,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:17,987 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:17,993 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,993 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:17,994 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:17,994 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:17,994 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:17,997 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,997 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:17,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cfb423[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:17,998 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:17,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,000 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,001 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,003 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,003 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,005 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,005 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,005 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,013 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,014 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,014 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:18,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,013 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,017 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e5349669ea814243befde008445866f1 2023-03-23 19:56:18,114 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,117 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,117 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,118 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,119 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,122 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,123 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,124 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d37ae83[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,124 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,126 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,126 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,128 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,130 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,130 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,130 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,132 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,133 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,136 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,136 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,136 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,136 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,236 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,240 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,241 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,243 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438ee437[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,249 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,249 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,250 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,252 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,252 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601438253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601438253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,255 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,255 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,255 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,255 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,255 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:18,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601438255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,258 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,258 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,258 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,259 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,260 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,262 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,263 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,263 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,363 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,367 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,370 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645d49c6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,377 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,378 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,379 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,379 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,382 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,382 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,383 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,383 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,384 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,384 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,386 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,388 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,389 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,394 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,394 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/5195987f6b0b4766aa9e79764f645ac4 2023-03-23 19:56:18,494 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,501 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,501 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,501 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,501 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,506 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,506 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,508 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,524 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,525 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,527 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,527 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d0a87c2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,531 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,531 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,532 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,533 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,538 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,538 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,538 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,540 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,542 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,543 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,547 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,547 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,547 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,547 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,647 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,651 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,651 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,652 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,652 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,653 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,653 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,653 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,653 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,655 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,655 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,655 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,655 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,655 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,657 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,657 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,658 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,658 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de97956[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,659 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,659 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,661 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,661 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,661 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,661 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,661 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,661 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,662 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,662 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,664 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,665 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,665 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,667 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,667 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,667 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,668 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,669 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,675 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,676 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,676 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,676 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,676 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,776 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,782 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,782 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,783 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,783 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,784 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,784 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,785 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,785 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,786 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,786 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,786 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:18,786 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:18,786 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:18,788 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,789 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,789 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f29580[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:18,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:18,790 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,791 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:18,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:18,792 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:18,792 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,792 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:18,795 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,796 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:18,798 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:18,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:18,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:18,799 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:18,799 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:18,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,800 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:18,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,801 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,803 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:18,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,805 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:18,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:18,806 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:18,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,806 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:18,806 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:18,806 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:18,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:18,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/2c15a5b3cd6b4e11900ab55c1b1df4ce 2023-03-23 19:56:18,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e5349669ea814243befde008445866f1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1 2023-03-23 19:56:18,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1, entries=200, sequenceid=110, filesize=31.9 K 2023-03-23 19:56:18,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/5195987f6b0b4766aa9e79764f645ac4 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4 2023-03-23 19:56:18,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4, entries=150, sequenceid=110, filesize=11.6 K 2023-03-23 19:56:18,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/2c15a5b3cd6b4e11900ab55c1b1df4ce as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce 2023-03-23 19:56:18,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce, entries=150, sequenceid=110, filesize=11.6 K 2023-03-23 19:56:18,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 92074693d0baf272335970fad69a35c6 in 1771ms, sequenceid=110, compaction requested=false 2023-03-23 19:56:18,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:18,906 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:18,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:18,912 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:18,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:18,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:18,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,912 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,914 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:18,914 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:18,914 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:18,915 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:18,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:18,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:18,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:18,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:18,924 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:18,932 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:18,970 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323d50f7c431bfb4554a32375b7f198a5ae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d50f7c431bfb4554a32375b7f198a5ae 2023-03-23 19:56:18,977 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/6fd393b144e5424d886c47290713feb8 2023-03-23 19:56:18,998 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/7396d94e9bab4ce1b35b540ae383fa33 2023-03-23 19:56:19,020 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5291bb782fde4ae8a8814ef6c8de96fd 2023-03-23 19:56:19,025 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/6fd393b144e5424d886c47290713feb8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8 2023-03-23 19:56:19,028 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8, entries=150, sequenceid=133, filesize=25.2 K 2023-03-23 19:56:19,030 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/7396d94e9bab4ce1b35b540ae383fa33 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33 2023-03-23 19:56:19,045 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33, entries=150, sequenceid=133, filesize=11.7 K 2023-03-23 19:56:19,046 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5291bb782fde4ae8a8814ef6c8de96fd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd 2023-03-23 19:56:19,050 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd, entries=150, sequenceid=133, filesize=11.7 K 2023-03-23 19:56:19,051 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 92074693d0baf272335970fad69a35c6 in 119ms, sequenceid=133, compaction requested=true 2023-03-23 19:56:19,051 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:19,051 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,056 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,057 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476ff601[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,059 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,060 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,060 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,060 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,062 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,062 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,062 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,062 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,064 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,064 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,065 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,065 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,067 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,068 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,068 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,070 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,072 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,072 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,072 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,073 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,072 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,073 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,173 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,177 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,177 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,179 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,179 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,181 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,182 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:19,183 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,185 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,186 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e524db9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,187 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,190 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,190 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,190 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,191 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,194 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,194 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,196 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,196 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,196 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,196 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,198 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,200 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,201 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,201 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:19,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:56:19,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:19,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:19,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:19,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:19,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:19,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:19,300 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323b362606dd18142bea570e9b2c38354f5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b362606dd18142bea570e9b2c38354f5 2023-03-23 19:56:19,301 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,306 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,307 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,310 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,310 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601379307, max timestamp=1679601379315Num uniques 859; 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:19,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601379307, max timestamp=1679601379315Num uniques 859; 2023-03-23 19:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601379307, max timestamp=1679601379315Num uniques 859; 2023-03-23 19:56:19,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,317 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,319 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,319 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c7beec[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,321 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,323 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,323 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/d378a905a1ad447eada8fa053c9e1bd6 2023-03-23 19:56:19,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,329 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,329 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,333 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,337 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,337 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601439339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601439342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601439343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,437 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,440 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,440 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,441 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,442 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,443 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,444 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,444 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,444 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,444 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,446 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601439444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25447ae1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601439445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601439448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,450 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,457 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,457 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,460 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,464 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,464 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,465 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,464 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,565 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,571 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,571 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,574 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,574 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,579 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d234fb6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,584 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,586 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,586 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,590 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,596 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,596 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,595 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,596 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,595 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601439649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601439651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601439651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,696 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,699 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,703 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,703 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,705 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4140f62e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,708 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,714 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,714 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,716 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,721 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,721 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,721 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/00b0fcce60a34cc796c1a27b7d1e9ec0 2023-03-23 19:56:19,821 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,825 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,827 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,831 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4351034[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,846 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,847 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,848 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,848 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,849 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,851 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,851 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,851 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,851 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,851 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,856 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,857 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,860 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,860 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,860 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,860 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601439953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601439955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601439957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,960 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:19,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:19,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:19,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:19,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:19,963 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:19,963 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:19,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:19,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:19,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:19,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:19,966 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:19,967 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,967 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:19,967 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:19,967 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:19,968 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220f121[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:19,970 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:19,970 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,973 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:19,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,973 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,974 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:19,976 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:19,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:19,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:19,977 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:19,977 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:19,978 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,978 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,978 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,982 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:19,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:19,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:19,982 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:19,982 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:19,982 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:19,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:19,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601439987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:19,990 DEBUG [Thread-3436] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:56:20,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601439998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,002 DEBUG [Thread-3440] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:56:20,082 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:20,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:20,086 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:20,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:20,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:20,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:20,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,093 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:20,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:20,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:20,095 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,095 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:20,095 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:20,095 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,095 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:20,096 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:20,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:20,098 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579402c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:20,098 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,100 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:20,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,100 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:20,100 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:20,101 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:20,103 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:20,104 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:20,106 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,106 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,106 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:20,106 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,108 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:20,110 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:20,110 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,110 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:20,110 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,111 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:20,111 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/a8a204dbcfc64af090d2c1a3ee41ef51 2023-03-23 19:56:20,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/d378a905a1ad447eada8fa053c9e1bd6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6 2023-03-23 19:56:20,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6, entries=350, sequenceid=144, filesize=52.6 K 2023-03-23 19:56:20,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/00b0fcce60a34cc796c1a27b7d1e9ec0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0 2023-03-23 19:56:20,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0, entries=150, sequenceid=144, filesize=11.8 K 2023-03-23 19:56:20,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/a8a204dbcfc64af090d2c1a3ee41ef51 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51 2023-03-23 19:56:20,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51, entries=150, sequenceid=144, filesize=11.8 K 2023-03-23 19:56:20,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 92074693d0baf272335970fad69a35c6 in 885ms, sequenceid=144, compaction requested=true 2023-03-23 19:56:20,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:20,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:20,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:20,170 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:20,170 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:20,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:20,171 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138226 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:20,171 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:20,171 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:20,171 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:20,171 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,171 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,171 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09712bb24cb840b1af26625b62251cb2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=135.0 K 2023-03-23 19:56:20,171 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/be09fc6cd0f7495489bf63a61e22abfc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=46.8 K 2023-03-23 19:56:20,172 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 09712bb24cb840b1af26625b62251cb2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374721 2023-03-23 19:56:20,172 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting be09fc6cd0f7495489bf63a61e22abfc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374721 2023-03-23 19:56:20,172 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e5349669ea814243befde008445866f1, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601375841 2023-03-23 19:56:20,172 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5195987f6b0b4766aa9e79764f645ac4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601375841 2023-03-23 19:56:20,173 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6fd393b144e5424d886c47290713feb8, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679601377124 2023-03-23 19:56:20,173 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7396d94e9bab4ce1b35b540ae383fa33, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679601377124 2023-03-23 19:56:20,173 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting d378a905a1ad447eada8fa053c9e1bd6, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379272 2023-03-23 19:56:20,173 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 00b0fcce60a34cc796c1a27b7d1e9ec0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379278 2023-03-23 19:56:20,179 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:20,184 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#492 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:20,186 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/1c17c185f8bd459bab6facfafde8f563 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1c17c185f8bd459bab6facfafde8f563 2023-03-23 19:56:20,191 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 1c17c185f8bd459bab6facfafde8f563(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:20,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:20,191 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=12, startTime=1679601380170; duration=0sec 2023-03-23 19:56:20,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:20,191 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-23 19:56:20,192 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-23 19:56:20,192 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:20,192 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,192 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7ee37bc14f74a53a62e165890e3d5bb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=46.8 K 2023-03-23 19:56:20,193 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c7ee37bc14f74a53a62e165890e3d5bb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679601374721 2023-03-23 19:56:20,193 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 2c15a5b3cd6b4e11900ab55c1b1df4ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679601375841 2023-03-23 19:56:20,193 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5291bb782fde4ae8a8814ef6c8de96fd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679601377124 2023-03-23 19:56:20,194 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting a8a204dbcfc64af090d2c1a3ee41ef51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379278 2023-03-23 19:56:20,201 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:20,202 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/11843d5ac0e142b49b9da72791026786 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/11843d5ac0e142b49b9da72791026786 2023-03-23 19:56:20,207 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 11843d5ac0e142b49b9da72791026786(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:20,207 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:20,207 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=12, startTime=1679601380170; duration=0sec 2023-03-23 19:56:20,207 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:20,209 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/d18eb2c9ce574beba69c3b16d770b71c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d18eb2c9ce574beba69c3b16d770b71c 2023-03-23 19:56:20,210 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:20,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:20,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:20,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:20,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:20,213 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:20,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:20,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:20,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:20,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:20,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:20,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:20,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:20,217 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into d18eb2c9ce574beba69c3b16d770b71c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:20,217 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:20,217 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=12, startTime=1679601380170; duration=0sec 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:20,217 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:20,217 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:20,217 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:20,240 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032312a96aa0ed134fd584cb3a4feaa6f939 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032312a96aa0ed134fd584cb3a4feaa6f939 2023-03-23 19:56:20,244 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e98a6bf2a0944a2285b147f5f957b834 2023-03-23 19:56:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:20,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601440465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601440466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601440468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601440569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601440569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601440571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,654 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/00ab48aa2eb0470fb54b489333e1f4a3 2023-03-23 19:56:20,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601440772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601440772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:20,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:20,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601440774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,064 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5a07aff26af04f36bf1ff782ebb66aba 2023-03-23 19:56:21,067 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e98a6bf2a0944a2285b147f5f957b834 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834 2023-03-23 19:56:21,071 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834, entries=150, sequenceid=183, filesize=25.3 K 2023-03-23 19:56:21,071 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/00ab48aa2eb0470fb54b489333e1f4a3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3 2023-03-23 19:56:21,075 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3, entries=150, sequenceid=183, filesize=11.8 K 2023-03-23 19:56:21,076 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5a07aff26af04f36bf1ff782ebb66aba as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba 2023-03-23 19:56:21,079 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba, entries=150, sequenceid=183, filesize=11.8 K 2023-03-23 19:56:21,080 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 92074693d0baf272335970fad69a35c6 in 863ms, sequenceid=183, compaction requested=false 2023-03-23 19:56:21,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:21,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,081 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:21,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:21,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ce272b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,086 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,087 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,087 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,091 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,092 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,092 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,094 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,095 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:21,103 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,103 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,104 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,104 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,104 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679601381097, max timestamp=1679601381116Num uniques 859; 2023-03-23 19:56:21,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679601381097, max timestamp=1679601381116Num uniques 859; 2023-03-23 19:56:21,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-23 19:56:21,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-23 19:56:21,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679601381097, max timestamp=1679601381116Num uniques 859; 2023-03-23 19:56:21,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-23 19:56:21,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601441133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601441133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601441134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,204 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,207 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,207 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,208 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,209 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,210 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,212 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd63bd6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,221 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,221 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,222 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,223 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,224 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,227 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,227 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,227 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,227 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,228 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,228 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,228 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601441236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601441236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601441237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,327 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,330 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,332 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,334 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,334 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,334 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,334 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,336 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,336 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,337 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6afa66f4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,339 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,340 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,341 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,343 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,344 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,344 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,345 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,346 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,347 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,350 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,350 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,351 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,351 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601441439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601441439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601441439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,451 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,454 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,456 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,456 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,457 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,459 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cd23a7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,462 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,462 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,467 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,467 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,467 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,471 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,475 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,475 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,475 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:21,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,506 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323b1bd8c1b8eb84beb9799c17dbf191905 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b1bd8c1b8eb84beb9799c17dbf191905 2023-03-23 19:56:21,511 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/6cc744d5b0a241ca89128ab9996ffc7f 2023-03-23 19:56:21,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/06055cc5d2ad476cbe42b3d0b851d8bc 2023-03-23 19:56:21,575 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,578 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,579 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,581 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,583 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,583 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,583 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,583 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,585 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1788bf4a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,586 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,587 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,588 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,589 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,589 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,591 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,591 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,594 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,594 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,594 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,596 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,598 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,598 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,598 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,598 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,699 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,702 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,705 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,705 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,706 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495e3ee6[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,715 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,715 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,716 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,717 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,720 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:21,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,721 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,721 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,720 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,721 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601441742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601441743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601441744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,821 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,824 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,827 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,828 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:21,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70aa2cff[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:21,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:21,832 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:21,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:21,834 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:21,836 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:21,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:21,836 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:21,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:21,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:21,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,840 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:21,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:21,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:21,843 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:21,843 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:21,843 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:21,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/29280553a3524efc92baba37fe6801a9 2023-03-23 19:56:21,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/6cc744d5b0a241ca89128ab9996ffc7f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f 2023-03-23 19:56:21,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f, entries=150, sequenceid=194, filesize=25.3 K 2023-03-23 19:56:21,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/06055cc5d2ad476cbe42b3d0b851d8bc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc 2023-03-23 19:56:21,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc, entries=100, sequenceid=194, filesize=9.4 K 2023-03-23 19:56:21,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/29280553a3524efc92baba37fe6801a9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9 2023-03-23 19:56:21,943 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:21,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:21,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:21,946 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:21,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:21,946 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:21,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,947 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,948 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:21,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:21,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9, entries=100, sequenceid=194, filesize=9.4 K 2023-03-23 19:56:21,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:21,948 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:21,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:21,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:21,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92074693d0baf272335970fad69a35c6 in 867ms, sequenceid=194, compaction requested=true 2023-03-23 19:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:21,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:21,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:21,949 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:21,950 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:21,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:21,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:21,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:21,950 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:21,950 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:21,951 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:21,951 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:21,960 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,960 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:21,960 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/11843d5ac0e142b49b9da72791026786, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=76.1 K 2023-03-23 19:56:21,960 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:21,960 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,960 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1c17c185f8bd459bab6facfafde8f563, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=33.2 K 2023-03-23 19:56:21,961 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 1c17c185f8bd459bab6facfafde8f563, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379278 2023-03-23 19:56:21,961 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 11843d5ac0e142b49b9da72791026786, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379278 2023-03-23 19:56:21,961 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 00ab48aa2eb0470fb54b489333e1f4a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679601379337 2023-03-23 19:56:21,961 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e98a6bf2a0944a2285b147f5f957b834, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679601379337 2023-03-23 19:56:21,961 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 06055cc5d2ad476cbe42b3d0b851d8bc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601380467 2023-03-23 19:56:21,962 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 6cc744d5b0a241ca89128ab9996ffc7f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601380464 2023-03-23 19:56:21,973 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:21,976 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#502 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:21,981 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/9e521e1ad94c406cbd002c6d28ba349b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/9e521e1ad94c406cbd002c6d28ba349b 2023-03-23 19:56:21,985 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 9e521e1ad94c406cbd002c6d28ba349b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:21,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:21,985 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=13, startTime=1679601381949; duration=0sec 2023-03-23 19:56:21,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:21,985 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:21,986 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:21,987 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:21,987 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:21,987 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d18eb2c9ce574beba69c3b16d770b71c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=33.2 K 2023-03-23 19:56:21,987 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d18eb2c9ce574beba69c3b16d770b71c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679601379278 2023-03-23 19:56:21,987 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5a07aff26af04f36bf1ff782ebb66aba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679601379337 2023-03-23 19:56:21,988 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 29280553a3524efc92baba37fe6801a9, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601380467 2023-03-23 19:56:21,993 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:22,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:22,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:22,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601442254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601442254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601442256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601442358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601442358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601442358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,377 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032355f297373cb042d899ab063f3f3adaa2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032355f297373cb042d899ab063f3f3adaa2 2023-03-23 19:56:22,394 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/10ae53457728457085ff5d7d31d81e39 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/10ae53457728457085ff5d7d31d81e39 2023-03-23 19:56:22,398 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 10ae53457728457085ff5d7d31d81e39(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:22,398 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:22,398 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=13, startTime=1679601381949; duration=0sec 2023-03-23 19:56:22,398 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:22,400 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/d99a2345122a4b3583ac95bb651e0722 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d99a2345122a4b3583ac95bb651e0722 2023-03-23 19:56:22,403 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into d99a2345122a4b3583ac95bb651e0722(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:22,403 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:22,404 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=13, startTime=1679601381950; duration=0sec 2023-03-23 19:56:22,404 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:22,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601442560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601442561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601442561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,781 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/a329a16f29924491885107a073eb74d1 2023-03-23 19:56:22,790 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4f10cf2547a64dd8a357ed580a48b070 2023-03-23 19:56:22,798 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/9da37a06f4514a7487ebe9cde61b1ece 2023-03-23 19:56:22,801 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/a329a16f29924491885107a073eb74d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1 2023-03-23 19:56:22,804 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1, entries=150, sequenceid=228, filesize=25.3 K 2023-03-23 19:56:22,805 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4f10cf2547a64dd8a357ed580a48b070 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070 2023-03-23 19:56:22,808 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070, entries=150, sequenceid=228, filesize=11.8 K 2023-03-23 19:56:22,812 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/9da37a06f4514a7487ebe9cde61b1ece as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece 2023-03-23 19:56:22,815 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece, entries=150, sequenceid=228, filesize=11.8 K 2023-03-23 19:56:22,816 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 866ms, sequenceid=228, compaction requested=false 2023-03-23 19:56:22,816 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:22,816 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:22,821 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,822 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,822 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:22,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb6f956[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:22,823 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:22,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,824 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:22,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:22,825 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:22,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:22,826 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:22,828 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:22,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:22,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:22,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:22,828 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:22,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:22,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,832 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:22,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:22,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:22,836 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:22,836 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:22,835 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:22,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:22,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:22,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:22,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:22,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601442882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601442883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601442883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,936 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:22,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:22,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:22,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:22,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:22,939 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,939 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,943 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:22,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:22,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:22,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:22,945 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:22,945 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:22,945 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:22,945 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:22,945 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:22,947 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:22,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e966365[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:22,949 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:22,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,951 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:22,956 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:22,958 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:22,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:22,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:22,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:22,958 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:22,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:22,961 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:22,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,962 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:22,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:22,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:22,965 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:22,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:22,965 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:22,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:22,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:22,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:22,966 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:22,966 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:22,966 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:22,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601442984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601442986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:22,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:22,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601442986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,066 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,069 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,071 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,081 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,081 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,082 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,082 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,082 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,082 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,085 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,086 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5607c95f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,087 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,089 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,091 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,091 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,091 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,093 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,094 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,095 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,097 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,098 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:23,098 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,100 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,100 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601443188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601443188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601443189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,200 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,203 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,204 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,206 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,206 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,206 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,206 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,208 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@776bdab4[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,209 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,215 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,216 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,218 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,219 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,219 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,220 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,220 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,220 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,221 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,222 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,224 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,225 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,225 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,225 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,225 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,225 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,224 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,225 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:23,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,277 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032377e92526016441a98ce1040098b5361f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032377e92526016441a98ce1040098b5361f 2023-03-23 19:56:23,282 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/7ff95642f8aa4aeb98f3f2f7cf35720e 2023-03-23 19:56:23,325 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,328 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,332 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,332 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,332 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,333 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,334 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c9cf146[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,339 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,339 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,340 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,340 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,341 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,343 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,343 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,343 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,347 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,348 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,349 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,351 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,351 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,352 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,352 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,352 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,352 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,352 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,352 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,452 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,456 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,457 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,459 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cc38fb[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,463 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,465 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,465 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,466 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,469 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,469 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,469 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,470 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,471 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,472 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,475 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,476 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,476 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,476 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,476 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601443490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601443491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601443492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,576 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,580 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,582 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,587 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e950ac[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,589 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,590 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,590 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,590 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,590 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,592 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,594 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,594 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,599 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,600 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,600 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,601 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,604 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,604 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,603 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,603 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,604 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/8a4cbd58e60348ba99921e9b6e25995f 2023-03-23 19:56:23,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/dae000bd25cf49308f313beefccba3d1 2023-03-23 19:56:23,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/7ff95642f8aa4aeb98f3f2f7cf35720e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e 2023-03-23 19:56:23,704 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e, entries=200, sequenceid=244, filesize=32.1 K 2023-03-23 19:56:23,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/8a4cbd58e60348ba99921e9b6e25995f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f 2023-03-23 19:56:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,707 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,709 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f, entries=150, sequenceid=244, filesize=11.8 K 2023-03-23 19:56:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/dae000bd25cf49308f313beefccba3d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:23,713 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,713 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1, entries=150, sequenceid=244, filesize=11.8 K 2023-03-23 19:56:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,714 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 92074693d0baf272335970fad69a35c6 in 849ms, sequenceid=244, compaction requested=true 2023-03-23 19:56:23,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:23,714 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:23,714 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:23,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,715 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:23,715 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:23,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fe8d5f[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:23,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:23,715 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:23,715 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,715 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,715 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:23,715 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/10ae53457728457085ff5d7d31d81e39, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=83.1 K 2023-03-23 19:56:23,715 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,716 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/9e521e1ad94c406cbd002c6d28ba349b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=35.6 K 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 10ae53457728457085ff5d7d31d81e39, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601379341 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9e521e1ad94c406cbd002c6d28ba349b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601379341 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting a329a16f29924491885107a073eb74d1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1679601381131 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4f10cf2547a64dd8a357ed580a48b070, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1679601381131 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 7ff95642f8aa4aeb98f3f2f7cf35720e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382253 2023-03-23 19:56:23,716 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 8a4cbd58e60348ba99921e9b6e25995f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382255 2023-03-23 19:56:23,717 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:23,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,718 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:23,719 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:23,722 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:23,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:23,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:23,722 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:23,722 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:23,722 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:23,724 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,724 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,724 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:23,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,725 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#511 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:23,726 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,727 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,730 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:23,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:23,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:23,730 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:23,730 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:23,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,730 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:23,731 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:23,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:23,830 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:23,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:23,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:23,834 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:23,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:23,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:23,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,840 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:23,840 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:23,840 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:23,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:23,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:23,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:23,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:23,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:23,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:23,842 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:56:23,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:23,842 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:23,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:23,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:23,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:23,843 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:23,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:23,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:24,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601444005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601444005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601444006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48320 deadline: 1679601444011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,015 DEBUG [Thread-3436] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:56:24,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48300 deadline: 1679601444021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,025 DEBUG [Thread-3440] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:56:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601444110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601444110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601444110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,137 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/3555567dd5b04c13af5729d3c8a991ae as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3555567dd5b04c13af5729d3c8a991ae 2023-03-23 19:56:24,140 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 3555567dd5b04c13af5729d3c8a991ae(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:24,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:24,140 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=13, startTime=1679601383714; duration=0sec 2023-03-23 19:56:24,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:24,140 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:24,141 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:24,141 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:24,141 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:24,141 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d99a2345122a4b3583ac95bb651e0722, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=35.6 K 2023-03-23 19:56:24,142 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d99a2345122a4b3583ac95bb651e0722, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679601379341 2023-03-23 19:56:24,142 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 9da37a06f4514a7487ebe9cde61b1ece, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1679601381131 2023-03-23 19:56:24,142 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting dae000bd25cf49308f313beefccba3d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382255 2023-03-23 19:56:24,149 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:24,170 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/4b0f4d96f49648ebacef15aef34f9cbf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/4b0f4d96f49648ebacef15aef34f9cbf 2023-03-23 19:56:24,174 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into 4b0f4d96f49648ebacef15aef34f9cbf(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:24,174 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:24,174 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=13, startTime=1679601383714; duration=0sec 2023-03-23 19:56:24,174 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:24,254 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303234c48fc12bb3146e3992a27634536f97d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234c48fc12bb3146e3992a27634536f97d 2023-03-23 19:56:24,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601444312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601444313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601444313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1679601444613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601444615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48284 deadline: 1679601444616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,662 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/58e5a1fcea0844dab40b9d83c2d55759 2023-03-23 19:56:24,670 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/da941e9c67af448e8ed57412846b6f76 2023-03-23 19:56:24,681 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5bf93a478eff47d9afa3636f5b6ad51c 2023-03-23 19:56:24,684 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/58e5a1fcea0844dab40b9d83c2d55759 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759 2023-03-23 19:56:24,687 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759, entries=150, sequenceid=263, filesize=25.4 K 2023-03-23 19:56:24,688 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/da941e9c67af448e8ed57412846b6f76 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76 2023-03-23 19:56:24,691 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76, entries=150, sequenceid=263, filesize=11.9 K 2023-03-23 19:56:24,692 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/5bf93a478eff47d9afa3636f5b6ad51c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c 2023-03-23 19:56:24,694 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c, entries=150, sequenceid=263, filesize=11.9 K 2023-03-23 19:56:24,695 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 92074693d0baf272335970fad69a35c6 in 853ms, sequenceid=263, compaction requested=false 2023-03-23 19:56:24,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:24,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:24,697 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7356c7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:24,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:24,699 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:24,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:24,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:24,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:24,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:24,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:24,704 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:24,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:24,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:24,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:24,704 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:24,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:24,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:24,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:24,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,709 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:24,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:24,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:24,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:24,711 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:24,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:24,711 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:24,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:24,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,811 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:24,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:24,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:24,814 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:24,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:24,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:24,814 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:24,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:24,816 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:24,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:24,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:24,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:24,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:24,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:24,818 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:24,818 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:24,828 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323eb6b14640d9744ff990a2740517c9205 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb6b14640d9744ff990a2740517c9205 2023-03-23 19:56:24,837 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/4edba96c1e4e4b208142e0c569f346a0 2023-03-23 19:56:24,845 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d070fbd03a0c4782a3a8df1945468a99 2023-03-23 19:56:24,941 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0 2023-03-23 19:56:24,945 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into da2b9b6ec1874a80b0ed0fd18c6d3ff0(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-23 19:56:24,945 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:24,945 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=13, startTime=1679601383714; duration=1sec 2023-03-23 19:56:24,945 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:25,117 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:25,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48260 deadline: 1679601445133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601445133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601445134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601445236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601445236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601445237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,262 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/81953243e2c14799994ed087c8b45b64 2023-03-23 19:56:25,266 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/4edba96c1e4e4b208142e0c569f346a0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0 2023-03-23 19:56:25,269 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0, entries=150, sequenceid=278, filesize=25.4 K 2023-03-23 19:56:25,269 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d070fbd03a0c4782a3a8df1945468a99 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99 2023-03-23 19:56:25,272 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99, entries=150, sequenceid=278, filesize=11.9 K 2023-03-23 19:56:25,273 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/81953243e2c14799994ed087c8b45b64 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64 2023-03-23 19:56:25,276 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64, entries=150, sequenceid=278, filesize=11.9 K 2023-03-23 19:56:25,276 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 92074693d0baf272335970fad69a35c6 in 458ms, sequenceid=278, compaction requested=true 2023-03-23 19:56:25,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:25,276 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:25,278 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:25,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:25,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb8a2b2[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:25,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:25,280 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,282 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:25,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:25,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:25,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:25,286 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:25,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:25,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:25,286 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:25,286 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:25,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:25,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,288 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:25,288 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,290 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:25,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:25,297 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:25,297 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,298 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:25,298 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,297 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:25,298 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:25,398 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:25,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:25,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:25,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:25,401 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:25,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:25,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:25,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:25,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,401 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:25,403 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:25,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:25,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:25,403 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:25,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:25,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:25,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:25,405 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:25,405 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:25,405 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:25,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:25,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:25,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601445448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601445449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601445450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601445552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601445552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601445553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601445756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601445757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:25,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601445757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:25,818 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032399be9bfd52a14f64a68645795d4f7eb5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032399be9bfd52a14f64a68645795d4f7eb5 2023-03-23 19:56:25,822 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/3aa7659192cf4d679fdba6f4cffafc30 2023-03-23 19:56:25,831 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/c2e4a60aaa69453fb7924faecbedd6d1 2023-03-23 19:56:26,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601446059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601446060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601446061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,239 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3cb682ba7f6a4dd6a5c43f071b863bfd 2023-03-23 19:56:26,243 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/3aa7659192cf4d679fdba6f4cffafc30 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30 2023-03-23 19:56:26,255 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30, entries=150, sequenceid=298, filesize=25.4 K 2023-03-23 19:56:26,255 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/c2e4a60aaa69453fb7924faecbedd6d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1 2023-03-23 19:56:26,259 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1, entries=150, sequenceid=298, filesize=11.9 K 2023-03-23 19:56:26,260 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3cb682ba7f6a4dd6a5c43f071b863bfd as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd 2023-03-23 19:56:26,263 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd, entries=150, sequenceid=298, filesize=11.9 K 2023-03-23 19:56:26,264 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 92074693d0baf272335970fad69a35c6 in 859ms, sequenceid=298, compaction requested=true 2023-03-23 19:56:26,264 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:26,264 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:26,269 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:26,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3857f959[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:26,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:26,271 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,273 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:26,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,273 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:26,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:26,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,276 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:26,276 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:26,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:26,279 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:26,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:26,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:26,279 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:26,279 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:26,280 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,280 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,280 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:26,281 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,282 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:26,284 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:26,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:26,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:26,284 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,285 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:26,285 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:26,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:26,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:26,385 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:26,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:26,388 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:26,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:26,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:26,388 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:26,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,389 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,390 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:26,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:26,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,390 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:26,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:26,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:26,391 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:26,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:26,391 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:26,392 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:26,392 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:26,402 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032368246333e3f941bf934ec09bff06e8a5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368246333e3f941bf934ec09bff06e8a5 2023-03-23 19:56:26,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:26,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:26,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601446584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601446585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601446586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601446688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601446691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601446691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,807 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/09945be1e71948099c2a92b283307afe 2023-03-23 19:56:26,816 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/0f90c0f42d9d45619a0a6dde04353c9e 2023-03-23 19:56:26,829 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/273f130cb12e47a9ba3ecf34e2f34c13 2023-03-23 19:56:26,835 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/09945be1e71948099c2a92b283307afe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe 2023-03-23 19:56:26,838 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe, entries=150, sequenceid=311, filesize=25.4 K 2023-03-23 19:56:26,840 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/0f90c0f42d9d45619a0a6dde04353c9e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e 2023-03-23 19:56:26,847 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e, entries=150, sequenceid=311, filesize=11.9 K 2023-03-23 19:56:26,848 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/273f130cb12e47a9ba3ecf34e2f34c13 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13 2023-03-23 19:56:26,852 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13, entries=150, sequenceid=311, filesize=11.9 K 2023-03-23 19:56:26,853 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 92074693d0baf272335970fad69a35c6 in 461ms, sequenceid=311, compaction requested=true 2023-03-23 19:56:26,853 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:26,853 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:26,855 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,856 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c570f79[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:26,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:26,858 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,859 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:26,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,859 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:26,861 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:26,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,861 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:26,864 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:26,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:26,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:26,864 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:26,864 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:26,866 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,866 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:26,867 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,867 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,868 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,873 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:26,873 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:26,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:26,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:26,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:26,874 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,874 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:26,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:26,874 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:26,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:26,897 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:26,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:26,909 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323198aea3e61084d9898612faf00710f0f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323198aea3e61084d9898612faf00710f0f 2023-03-23 19:56:26,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601446906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601446907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601446909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,974 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:26,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:26,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:26,978 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:26,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:26,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:26,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:26,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,978 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,979 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:26,979 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,980 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:26,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:26,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:26,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:26,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:26,981 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:26,983 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df3a15a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:26,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:26,985 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,986 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:26,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,986 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:26,988 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:26,990 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:26,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:26,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:26,991 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:26,991 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:26,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:26,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,992 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,993 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:26,994 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:26,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:26,999 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:26,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:26,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:26,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:26,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:26,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:26,999 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:27,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:27,000 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:27,000 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601447011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601447011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601447013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,100 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:27,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:27,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:27,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:27,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:27,103 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:27,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:27,103 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:27,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,104 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,109 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:27,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:27,109 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,110 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:27,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:27,111 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,111 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:27,111 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:27,111 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:27,111 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,113 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a03764[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:27,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:27,114 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,116 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,116 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:27,118 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:27,120 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:27,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:27,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:27,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:27,121 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:27,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,123 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,123 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,124 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,127 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,127 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,127 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:27,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601447213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601447214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601447217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,227 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:27,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:27,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:27,231 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:27,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:27,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:27,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,231 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,232 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,233 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:27,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:27,233 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,233 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:27,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:27,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:27,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:27,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:27,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:27,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:27,236 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:27,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:27,238 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343a8d78[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:27,240 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:27,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,240 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:27,242 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:27,244 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:27,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:27,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:27,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:27,244 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:27,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,245 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:27,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:27,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:27,246 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:27,247 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:27,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:27,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:27,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:27,249 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:27,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,250 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,250 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:27,250 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,250 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:27,250 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:27,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:27,314 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/2aabda79beec47ef8c6c3db291e2d1fe 2023-03-23 19:56:27,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/31634e14e5c447bdb65901585a03eebc 2023-03-23 19:56:27,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c6fb3a1a8bf7424384b0d6dacff7eaf9 2023-03-23 19:56:27,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/2aabda79beec47ef8c6c3db291e2d1fe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe 2023-03-23 19:56:27,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe, entries=150, sequenceid=332, filesize=25.4 K 2023-03-23 19:56:27,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/31634e14e5c447bdb65901585a03eebc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc 2023-03-23 19:56:27,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc, entries=150, sequenceid=332, filesize=11.9 K 2023-03-23 19:56:27,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c6fb3a1a8bf7424384b0d6dacff7eaf9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9 2023-03-23 19:56:27,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9, entries=150, sequenceid=332, filesize=11.9 K 2023-03-23 19:56:27,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 446ms, sequenceid=332, compaction requested=true 2023-03-23 19:56:27,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:27,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:27,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:27,343 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:56:27,344 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:56:27,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:27,345 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156554 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:56:27,345 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73394 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:56:27,345 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:27,345 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:27,345 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,345 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,345 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=152.9 K 2023-03-23 19:56:27,345 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3555567dd5b04c13af5729d3c8a991ae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=71.7 K 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting da2b9b6ec1874a80b0ed0fd18c6d3ff0, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382255 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3555567dd5b04c13af5729d3c8a991ae, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382255 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 58e5a1fcea0844dab40b9d83c2d55759, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679601382876 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting da941e9c67af448e8ed57412846b6f76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679601382876 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 4edba96c1e4e4b208142e0c569f346a0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601384001 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d070fbd03a0c4782a3a8df1945468a99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601384001 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 3aa7659192cf4d679fdba6f4cffafc30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679601385128 2023-03-23 19:56:27,346 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c2e4a60aaa69453fb7924faecbedd6d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679601385128 2023-03-23 19:56:27,347 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 09945be1e71948099c2a92b283307afe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679601385441 2023-03-23 19:56:27,347 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 0f90c0f42d9d45619a0a6dde04353c9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679601385441 2023-03-23 19:56:27,347 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 2aabda79beec47ef8c6c3db291e2d1fe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:27,347 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 31634e14e5c447bdb65901585a03eebc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:27,350 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:27,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:27,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:27,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:27,354 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:27,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:27,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:27,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,354 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,357 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,357 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#528 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:27,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,357 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:27,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:27,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:27,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:27,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:27,359 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:27,364 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,364 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:27,364 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:56:27,364 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:27,364 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:27,364 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:27,365 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:27,365 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:27,365 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:27,374 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#529 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:27,383 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/3c8a9b6ccf49463d9eb9d62b135cc21e as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3c8a9b6ccf49463d9eb9d62b135cc21e 2023-03-23 19:56:27,388 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 3c8a9b6ccf49463d9eb9d62b135cc21e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:27,388 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:27,388 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=10, startTime=1679601387343; duration=0sec 2023-03-23 19:56:27,388 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:27,388 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-23 19:56:27,390 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73394 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-23 19:56:27,390 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:27,390 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:27,390 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/4b0f4d96f49648ebacef15aef34f9cbf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=71.7 K 2023-03-23 19:56:27,390 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 4b0f4d96f49648ebacef15aef34f9cbf, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679601382255 2023-03-23 19:56:27,391 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 5bf93a478eff47d9afa3636f5b6ad51c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679601382876 2023-03-23 19:56:27,391 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 81953243e2c14799994ed087c8b45b64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679601384001 2023-03-23 19:56:27,391 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3cb682ba7f6a4dd6a5c43f071b863bfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679601385128 2023-03-23 19:56:27,392 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 273f130cb12e47a9ba3ecf34e2f34c13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679601385441 2023-03-23 19:56:27,392 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c6fb3a1a8bf7424384b0d6dacff7eaf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:27,394 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032359e93ae33721412f804988eda8bd7dcd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032359e93ae33721412f804988eda8bd7dcd 2023-03-23 19:56:27,403 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#531 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:27,409 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/f953deabb393418f93e885bd10f0c07c 2023-03-23 19:56:27,415 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/29524add966a40649a9d7d07b3f91a60 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29524add966a40649a9d7d07b3f91a60 2023-03-23 19:56:27,419 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into 29524add966a40649a9d7d07b3f91a60(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:27,419 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:27,420 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=10, startTime=1679601387344; duration=0sec 2023-03-23 19:56:27,420 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:27,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:27,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601447538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601447538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601387533, max timestamp=1679601387540Num uniques 859; 2023-03-23 19:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601387533, max timestamp=1679601387540Num uniques 859; 2023-03-23 19:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601387533, max timestamp=1679601387540Num uniques 859; 2023-03-23 19:56:27,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:27,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601447585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601447642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601447642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601447689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,822 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/59f939a9f3ef400ca7dee4df9c754ff0 2023-03-23 19:56:27,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601447844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601447844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:27,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601447900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601448146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601448147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601448203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,230 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3ddd99300f3c47a183decd5e27860298 2023-03-23 19:56:28,234 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/f953deabb393418f93e885bd10f0c07c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c 2023-03-23 19:56:28,237 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c, entries=150, sequenceid=343, filesize=25.4 K 2023-03-23 19:56:28,238 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/59f939a9f3ef400ca7dee4df9c754ff0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0 2023-03-23 19:56:28,241 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0, entries=150, sequenceid=343, filesize=11.9 K 2023-03-23 19:56:28,241 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/3ddd99300f3c47a183decd5e27860298 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298 2023-03-23 19:56:28,244 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298, entries=150, sequenceid=343, filesize=11.9 K 2023-03-23 19:56:28,245 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92074693d0baf272335970fad69a35c6 in 881ms, sequenceid=343, compaction requested=false 2023-03-23 19:56:28,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:28,245 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:28,247 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20827041[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:28,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:28,248 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:28,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:28,251 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:28,253 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:28,253 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:28,253 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:28,253 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:28,253 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:28,256 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,256 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,256 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:28,257 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:28,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,263 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:28,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:28,264 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:28,263 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,264 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:28,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:28,364 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:28,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:28,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:28,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:28,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:28,367 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:28,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:28,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:28,367 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:28,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:28,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:28,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,369 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:28,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:28,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:28,370 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:28,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:28,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:28,371 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:28,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:28,382 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032358b7cbbac57f410eb907cd66b3804f71 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358b7cbbac57f410eb907cd66b3804f71 2023-03-23 19:56:28,387 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e458d3efbecd45ec94ff49dbe2a494e1 2023-03-23 19:56:28,395 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/1d06205127954d0dbd193a0a4e06ca1d 2023-03-23 19:56:28,595 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/b1bb745e8b384d90b2bc5534a78b4050 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/b1bb745e8b384d90b2bc5534a78b4050 2023-03-23 19:56:28,598 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into b1bb745e8b384d90b2bc5534a78b4050(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-23 19:56:28,598 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:28,598 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=10, startTime=1679601387343; duration=1sec 2023-03-23 19:56:28,598 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:28,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:28,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601448663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601448664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601448708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601448766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:28,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601448767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,803 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/31f12e0389e847339f34a623f5f3fe80 2023-03-23 19:56:28,807 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/e458d3efbecd45ec94ff49dbe2a494e1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1 2023-03-23 19:56:28,810 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1, entries=150, sequenceid=380, filesize=25.4 K 2023-03-23 19:56:28,811 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/1d06205127954d0dbd193a0a4e06ca1d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d 2023-03-23 19:56:28,814 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d, entries=150, sequenceid=380, filesize=11.9 K 2023-03-23 19:56:28,815 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/31f12e0389e847339f34a623f5f3fe80 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80 2023-03-23 19:56:28,818 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80, entries=150, sequenceid=380, filesize=11.9 K 2023-03-23 19:56:28,819 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 448ms, sequenceid=380, compaction requested=true 2023-03-23 19:56:28,819 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:28,819 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:28,824 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70abfc7a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:28,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:28,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,827 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:28,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,827 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:28,828 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:28,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,829 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:28,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:28,831 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:28,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:28,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:28,831 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:28,831 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:28,833 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,833 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:28,833 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:28,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:28,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,835 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:28,838 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:28,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:28,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:28,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,838 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:28,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:28,838 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:28,838 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:28,939 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:28,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:28,942 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:28,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:28,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:28,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:28,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,943 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,944 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,944 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:28,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:28,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,945 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:28,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:28,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:28,947 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:28,947 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:28,947 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:28,947 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:28,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:28,960 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323b73b1155d8194dc7ab71042c2da6f870 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b73b1155d8194dc7ab71042c2da6f870 2023-03-23 19:56:28,965 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/9a2c5fa7f7fd4a578413590319482305 2023-03-23 19:56:28,974 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/975c52a8ce99410b99b3294b71c22dbf 2023-03-23 19:56:28,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:28,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:28,982 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c7f0a39abb5a43cfb772d5eb3ea61821 2023-03-23 19:56:28,986 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/9a2c5fa7f7fd4a578413590319482305 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305 2023-03-23 19:56:28,990 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305, entries=100, sequenceid=392, filesize=18.6 K 2023-03-23 19:56:28,990 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/975c52a8ce99410b99b3294b71c22dbf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf 2023-03-23 19:56:28,994 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf, entries=100, sequenceid=392, filesize=9.5 K 2023-03-23 19:56:28,995 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c7f0a39abb5a43cfb772d5eb3ea61821 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821 2023-03-23 19:56:28,998 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821, entries=100, sequenceid=392, filesize=9.5 K 2023-03-23 19:56:28,998 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=60.38 KB/61830 for 92074693d0baf272335970fad69a35c6 in 51ms, sequenceid=392, compaction requested=true 2023-03-23 19:56:28,998 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:28,998 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,000 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,000 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:29,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4202ee1c[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,003 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,004 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,004 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,004 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,007 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,009 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,009 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,010 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,011 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,012 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,014 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,014 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,015 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,015 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,015 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,037 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032308d372abde7e4dd4bb994a6a4bbec7d8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032308d372abde7e4dd4bb994a6a4bbec7d8 2023-03-23 19:56:29,042 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/72efc34bc4954f23863ad44fa78daefe 2023-03-23 19:56:29,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601449040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601449042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,115 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,119 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,119 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,120 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,120 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,121 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,122 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,122 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,122 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,122 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,124 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2233fa69[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,126 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,127 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,127 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,127 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,129 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,129 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,132 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,132 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,133 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,133 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,133 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,135 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,137 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,139 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,139 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,140 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,140 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601449143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601449144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,240 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,243 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,243 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,244 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,244 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,245 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,246 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,246 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,246 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,246 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,249 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11008616[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,250 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,254 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,254 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,256 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,258 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,258 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,258 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,260 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,260 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,260 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,261 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,262 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,264 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,264 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,264 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,265 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,265 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,265 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,265 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601449346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601449348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,365 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,368 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,368 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,369 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,370 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,371 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,371 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,373 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee7e060[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,375 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,376 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,377 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,377 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,377 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,378 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,380 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,380 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,380 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,382 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,382 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,383 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,385 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,386 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,386 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:29,386 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,386 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/f2f7a276b2ed48638ee47b6b2594b2ed 2023-03-23 19:56:29,486 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,489 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,489 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,490 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,490 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,492 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,492 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,492 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,492 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,493 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,493 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,494 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,494 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,494 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,495 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bc08bae[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,497 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,497 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,498 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,499 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,499 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,500 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,502 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,502 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,502 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,505 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,505 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,505 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,505 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,507 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,521 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,521 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:29,522 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,522 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,622 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,625 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,625 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,627 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,627 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,629 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,631 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@569dc40d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,633 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,635 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,635 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,635 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,635 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,637 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,639 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,640 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,640 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,645 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,646 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,648 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,648 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,648 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,648 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:29,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601449650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601449650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:29,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601449720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,748 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,751 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,751 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,754 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,754 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,756 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,756 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,756 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,756 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,757 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b8c8fe[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,759 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,760 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,760 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,762 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,764 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,764 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,764 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,764 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,764 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,767 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,767 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,768 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,770 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,771 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,771 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:29,771 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,771 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/562a9712e9b543ffa8631c6ad11a7842 2023-03-23 19:56:29,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/72efc34bc4954f23863ad44fa78daefe as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe 2023-03-23 19:56:29,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe, entries=350, sequenceid=405, filesize=52.9 K 2023-03-23 19:56:29,871 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:29,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/f2f7a276b2ed48638ee47b6b2594b2ed as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed 2023-03-23 19:56:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:29,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:29,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed, entries=100, sequenceid=405, filesize=9.5 K 2023-03-23 19:56:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:29,875 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:29,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:29,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:29,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/562a9712e9b543ffa8631c6ad11a7842 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842 2023-03-23 19:56:29,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,875 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,877 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,877 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:29,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:29,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:29,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:29,879 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:29,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842, entries=100, sequenceid=405, filesize=9.5 K 2023-03-23 19:56:29,881 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 92074693d0baf272335970fad69a35c6 in 881ms, sequenceid=405, compaction requested=true 2023-03-23 19:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:29,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:29,881 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:56:29,881 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:56:29,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:29,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 152028 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56724 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:56:29,883 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:29,883 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/b1bb745e8b384d90b2bc5534a78b4050, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=148.5 K 2023-03-23 19:56:29,883 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,883 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3c8a9b6ccf49463d9eb9d62b135cc21e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=55.4 K 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting b1bb745e8b384d90b2bc5534a78b4050, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,883 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3c8a9b6ccf49463d9eb9d62b135cc21e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:29,884 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting f953deabb393418f93e885bd10f0c07c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1679601386905 2023-03-23 19:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ddc409[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:29,884 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 59f939a9f3ef400ca7dee4df9c754ff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1679601386905 2023-03-23 19:56:29,884 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:29,884 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting e458d3efbecd45ec94ff49dbe2a494e1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679601387578 2023-03-23 19:56:29,884 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,884 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 1d06205127954d0dbd193a0a4e06ca1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679601387578 2023-03-23 19:56:29,885 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 9a2c5fa7f7fd4a578413590319482305, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1679601388659 2023-03-23 19:56:29,885 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 975c52a8ce99410b99b3294b71c22dbf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1679601388659 2023-03-23 19:56:29,885 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 72efc34bc4954f23863ad44fa78daefe, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388984 2023-03-23 19:56:29,885 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f2f7a276b2ed48638ee47b6b2594b2ed, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388994 2023-03-23 19:56:29,887 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:29,887 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:29,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,887 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:29,889 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,889 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:29,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:29,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:29,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:29,892 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:29,892 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:29,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,894 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,895 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#544 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:29,896 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,896 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#543 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:29,898 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:29,899 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,902 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,902 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:29,902 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:29,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:29,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:29,902 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:29,902 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:29,903 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:29,903 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:29,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:29,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:29,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:29,903 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:29,912 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/f6b5dfb5d0aa4846bf7532de1a933d1f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f6b5dfb5d0aa4846bf7532de1a933d1f 2023-03-23 19:56:29,916 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into f6b5dfb5d0aa4846bf7532de1a933d1f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:29,916 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:29,916 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=11, startTime=1679601389881; duration=0sec 2023-03-23 19:56:29,917 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:29,917 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-23 19:56:29,918 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56724 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-23 19:56:29,918 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:29,918 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:29,918 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29524add966a40649a9d7d07b3f91a60, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=55.4 K 2023-03-23 19:56:29,919 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 29524add966a40649a9d7d07b3f91a60, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679601386583 2023-03-23 19:56:29,919 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 3ddd99300f3c47a183decd5e27860298, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1679601386905 2023-03-23 19:56:29,919 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 31f12e0389e847339f34a623f5f3fe80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679601387578 2023-03-23 19:56:29,919 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting c7f0a39abb5a43cfb772d5eb3ea61821, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1679601388659 2023-03-23 19:56:29,920 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 562a9712e9b543ffa8631c6ad11a7842, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388994 2023-03-23 19:56:29,929 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/5d862ecb361345f2b2398705e5e728cf as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5d862ecb361345f2b2398705e5e728cf 2023-03-23 19:56:29,929 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#545 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:29,933 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 5d862ecb361345f2b2398705e5e728cf(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:29,933 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:29,933 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=11, startTime=1679601389881; duration=0sec 2023-03-23 19:56:29,933 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:29,937 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/27791058d4a348d3aaf539685ef00c38 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/27791058d4a348d3aaf539685ef00c38 2023-03-23 19:56:29,940 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into 27791058d4a348d3aaf539685ef00c38(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:29,941 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:29,941 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=11, startTime=1679601389881; duration=0sec 2023-03-23 19:56:29,941 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:30,003 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,006 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,006 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,007 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,009 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,009 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,010 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,011 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,011 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,011 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:30,012 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,025 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323a0d644c7def641b2ad53e64c8e3cca43 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a0d644c7def641b2ad53e64c8e3cca43 2023-03-23 19:56:30,034 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/85427ae05cca4b449c0bec49d6dc5e72 2023-03-23 19:56:30,042 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/cf8f0116b02743b4b65c0eed5d86dcdc 2023-03-23 19:56:30,049 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/7a201b52895c434ab3f0b27331a79805 2023-03-23 19:56:30,053 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/85427ae05cca4b449c0bec49d6dc5e72 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72 2023-03-23 19:56:30,057 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72, entries=100, sequenceid=427, filesize=18.6 K 2023-03-23 19:56:30,058 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/cf8f0116b02743b4b65c0eed5d86dcdc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc 2023-03-23 19:56:30,062 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc, entries=100, sequenceid=427, filesize=9.5 K 2023-03-23 19:56:30,062 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/7a201b52895c434ab3f0b27331a79805 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805 2023-03-23 19:56:30,066 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805, entries=100, sequenceid=427, filesize=9.5 K 2023-03-23 19:56:30,068 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 92074693d0baf272335970fad69a35c6 in 57ms, sequenceid=427, compaction requested=false 2023-03-23 19:56:30,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:30,068 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,070 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dafba07[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,075 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,075 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,078 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,078 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,078 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,081 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,081 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,081 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,081 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,083 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,089 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,089 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,189 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,192 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,193 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,193 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:30,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-23 19:56:30,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:30,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,194 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:30,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:30,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,197 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,198 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,198 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,198 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,201 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,201 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15aaae13[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,203 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,204 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,204 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,206 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323cb38b7df5f4041eeb5831c076280c374 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323cb38b7df5f4041eeb5831c076280c374 2023-03-23 19:56:30,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,206 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,210 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,210 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,211 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=438, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/cc414a9b867f491ab4d94f931be47f6a 2023-03-23 19:56:30,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,212 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,214 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,219 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,219 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,218 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,218 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601390219, max timestamp=1679601390225Num uniques 859; 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601390219, max timestamp=1679601390225Num uniques 859; 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679601390219, max timestamp=1679601390225Num uniques 859; 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:30,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-23 19:56:30,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601450245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601450247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,319 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,322 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,323 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,324 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,324 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,328 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d07b60[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,329 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,331 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,331 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,333 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,335 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,335 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,337 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,339 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,342 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,342 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,342 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,342 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601450349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601450350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,443 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,446 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,446 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,448 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,448 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,449 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,449 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,449 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,450 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,454 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c0108ff[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,455 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,455 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,457 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,459 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,459 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,460 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,462 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,468 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,468 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,469 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,469 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,469 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601450551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601450551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,569 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,571 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,572 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,573 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,575 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,576 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,576 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,577 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c2ef2fa[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,578 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,579 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,580 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,582 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,585 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,585 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,586 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,587 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,591 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,591 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,591 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,591 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/76ac73c8e5134b29a54b3ffc307ec81c 2023-03-23 19:56:30,692 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,695 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,695 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,696 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,696 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,696 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,696 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,698 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c34135a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,702 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,704 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,704 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,706 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,708 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,708 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,710 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,710 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,710 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,710 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,711 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,714 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,714 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,714 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,715 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,715 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,715 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,815 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,818 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,818 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,822 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,823 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,824 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,825 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fac76c7[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,825 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,828 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,830 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,830 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,830 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,832 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,832 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,833 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,836 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,836 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,836 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:30,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601450854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601450856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,937 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:30,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:30,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:30,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:30,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:30,940 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:30,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:30,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:30,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,940 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,941 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,941 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,942 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:30,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:30,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:30,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:30,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:30,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:30,943 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:30,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:30,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:30,943 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:30,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:30,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:30,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:30,944 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:30,948 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bdec73d[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:30,950 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:30,950 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,953 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:30,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:30,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,954 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:30,954 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:30,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:30,957 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:30,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:30,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:30,957 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:30,957 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:30,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:30,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:30,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:30,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:30,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:30,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:30,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:30,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:30,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:30,964 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:30,965 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:31,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/b08c42066d8f464c839ee45fae7679eb 2023-03-23 19:56:31,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/cc414a9b867f491ab4d94f931be47f6a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a 2023-03-23 19:56:31,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a, entries=300, sequenceid=438, filesize=46.1 K 2023-03-23 19:56:31,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/76ac73c8e5134b29a54b3ffc307ec81c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c 2023-03-23 19:56:31,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c, entries=150, sequenceid=438, filesize=11.9 K 2023-03-23 19:56:31,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/b08c42066d8f464c839ee45fae7679eb as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb 2023-03-23 19:56:31,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb, entries=150, sequenceid=438, filesize=11.9 K 2023-03-23 19:56:31,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92074693d0baf272335970fad69a35c6 in 862ms, sequenceid=438, compaction requested=true 2023-03-23 19:56:31,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:31,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:31,057 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:31,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:31,057 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:31,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:31,057 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93066 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:31,058 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:31,058 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:31,058 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5d862ecb361345f2b2398705e5e728cf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=90.9 K 2023-03-23 19:56:31,058 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:31,058 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:31,058 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:31,058 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 5d862ecb361345f2b2398705e5e728cf, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388659 2023-03-23 19:56:31,058 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f6b5dfb5d0aa4846bf7532de1a933d1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=34.1 K 2023-03-23 19:56:31,059 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 85427ae05cca4b449c0bec49d6dc5e72, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679601389038 2023-03-23 19:56:31,059 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting f6b5dfb5d0aa4846bf7532de1a933d1f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388659 2023-03-23 19:56:31,059 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting cc414a9b867f491ab4d94f931be47f6a, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390180 2023-03-23 19:56:31,059 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting cf8f0116b02743b4b65c0eed5d86dcdc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679601389038 2023-03-23 19:56:31,059 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 76ac73c8e5134b29a54b3ffc307ec81c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390189 2023-03-23 19:56:31,065 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:31,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:31,068 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#553 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:31,072 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:31,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:31,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:31,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:31,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,072 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:31,073 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:31,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:31,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,074 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:31,074 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#552 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:31,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:31,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:31,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:31,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:31,080 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:31,081 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:31,081 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:31,089 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/35b5336a04264daab35170463314202a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/35b5336a04264daab35170463314202a 2023-03-23 19:56:31,094 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 35b5336a04264daab35170463314202a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:31,094 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:31,094 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=13, startTime=1679601391057; duration=0sec 2023-03-23 19:56:31,094 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:31,094 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:31,095 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:31,095 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:31,095 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:31,096 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/27791058d4a348d3aaf539685ef00c38, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=34.1 K 2023-03-23 19:56:31,096 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 27791058d4a348d3aaf539685ef00c38, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679601388659 2023-03-23 19:56:31,096 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7a201b52895c434ab3f0b27331a79805, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679601389038 2023-03-23 19:56:31,097 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting b08c42066d8f464c839ee45fae7679eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390189 2023-03-23 19:56:31,103 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:31,110 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/67a48ee9f8d04e10a6dc1020d894cb21 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/67a48ee9f8d04e10a6dc1020d894cb21 2023-03-23 19:56:31,115 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 67a48ee9f8d04e10a6dc1020d894cb21(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:31,115 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:31,115 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=13, startTime=1679601391057; duration=0sec 2023-03-23 19:56:31,115 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:31,115 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/7568cbef1c244966beb39c0e35210133 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7568cbef1c244966beb39c0e35210133 2023-03-23 19:56:31,118 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into 7568cbef1c244966beb39c0e35210133(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:31,119 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:31,119 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=13, startTime=1679601391057; duration=0sec 2023-03-23 19:56:31,119 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:31,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:31,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:31,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601451372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601451373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601451475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601451475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,513 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323772793a5ac0543a8abd0e1b64095932d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323772793a5ac0543a8abd0e1b64095932d 2023-03-23 19:56:31,518 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=473, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/480fd8f8575b40dc9ef981b1340480d8 2023-03-23 19:56:31,525 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d12e150a3d41440c93d351f1ae0b63d1 2023-03-23 19:56:31,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601451677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601451678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:31,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48284 deadline: 1679601451723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,727 DEBUG [Thread-3432] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., hostname=jenkins-hbase4.apache.org,34277,1679601218015, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-23 19:56:31,933 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/cea9185c292e4920a796b27023c819bc 2023-03-23 19:56:31,937 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/480fd8f8575b40dc9ef981b1340480d8 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8 2023-03-23 19:56:31,940 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8, entries=150, sequenceid=473, filesize=25.4 K 2023-03-23 19:56:31,941 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/d12e150a3d41440c93d351f1ae0b63d1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1 2023-03-23 19:56:31,944 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1, entries=150, sequenceid=473, filesize=11.9 K 2023-03-23 19:56:31,945 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/cea9185c292e4920a796b27023c819bc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc 2023-03-23 19:56:31,948 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc, entries=150, sequenceid=473, filesize=11.9 K 2023-03-23 19:56:31,948 INFO [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 92074693d0baf272335970fad69a35c6 in 868ms, sequenceid=473, compaction requested=false 2023-03-23 19:56:31,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:31,948 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:31,950 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:31,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:31,950 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107c5d9e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:31,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:31,952 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,953 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:31,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:31,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,953 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:31,953 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:31,955 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:31,955 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:31,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:31,956 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:31,958 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:31,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:31,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:31,958 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:31,958 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:31,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:31,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,959 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:31,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:31,960 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,961 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:31,964 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:31,963 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:31,964 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:31,964 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:31,964 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:31,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:31,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:32,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601452006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601452008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,064 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,067 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,067 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,067 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,067 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,068 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,068 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,069 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,069 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,070 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,071 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,071 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,073 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cde3208[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,075 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,075 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,076 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,076 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,077 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,078 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,080 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,080 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,080 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,080 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,080 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,082 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,082 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,082 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,083 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,084 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,088 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,088 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,088 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,089 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,089 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,089 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,089 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601452111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601452111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,189 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,192 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,192 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,194 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,194 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,195 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,196 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,196 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,196 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,196 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,196 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,197 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,198 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,198 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,199 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4098651a[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,200 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,205 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,205 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,206 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,207 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,210 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,210 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,210 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,211 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,211 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,211 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,212 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,213 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,215 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,216 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,215 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:32,216 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,216 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601452313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,316 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601452314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,319 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,319 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,322 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,322 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,323 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,323 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,323 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,323 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,323 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,325 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,325 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d82a51b[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,327 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,328 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,328 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,330 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,330 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,332 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,332 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,332 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,335 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,335 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,336 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,338 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,338 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,339 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,339 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:32,393 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303235031962b428b411fba967c2adba42bd5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235031962b428b411fba967c2adba42bd5 2023-03-23 19:56:32,439 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,442 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,442 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,443 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,444 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,444 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,445 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,446 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,447 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,448 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe08f98[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,449 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,451 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,451 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,451 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,452 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,452 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,453 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,455 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,455 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,456 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,457 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,457 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,459 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,461 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,461 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,462 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,462 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,462 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,562 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,565 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,565 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,567 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,568 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,568 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,569 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,569 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,571 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,571 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,572 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,572 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2243ba1e[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,573 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,574 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,574 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,575 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,575 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,579 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,579 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,581 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,581 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,581 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,583 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,583 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,585 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,588 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,588 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,589 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,589 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34277] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48260 deadline: 1679601452616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-23 19:56:32,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34277] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1679601452619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,689 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,692 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,692 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,693 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,693 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,693 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,694 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,695 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,695 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,698 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,698 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2170abf9[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,700 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,701 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,701 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,701 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,702 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,702 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,703 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,705 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,705 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,705 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,707 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,707 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,708 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,711 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,712 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,712 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,712 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,712 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,798 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=488, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/01f002db961042fb84c5b856ac0d87b0 2023-03-23 19:56:32,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/dc0b360cad154a6aa2c53d9dbb5c8bc9 2023-03-23 19:56:32,812 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:32,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-23 19:56:32,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-23 19:56:32,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-23 19:56:32,815 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-23 19:56:32,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-23 19:56:32,815 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-23 19:56:32,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,816 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,819 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-23 19:56:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,820 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-23 19:56:32,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-23 19:56:32,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-23 19:56:32,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-23 19:56:32,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. started... 2023-03-23 19:56:32,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. as already flushing 2023-03-23 19:56:32,821 DEBUG [rs(jenkins-hbase4.apache.org,34277,1679601218015)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-23 19:56:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,822 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-23 19:56:32,822 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-23 19:56:32,823 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,823 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,824 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-23 19:56:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34277,1679601218015' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-23 19:56:32,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-23 19:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97278b1[Count = 0] remaining members to acquire global barrier 2023-03-23 19:56:32,826 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,827 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-23 19:56:32,827 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-23 19:56:32,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,828 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-23 19:56:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34277,1679601218015' in zk 2023-03-23 19:56:32,830 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-23 19:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,34277,1679601218015' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-23 19:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/41cb1dc7948d418286f3520c8c950b0d 2023-03-23 19:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34277,1679601218015': 2023-03-23 19:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34277,1679601218015' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-23 19:56:32,834 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-23 19:56:32,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-23 19:56:32,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-23 19:56:32,834 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-23 19:56:32,834 INFO [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-23 19:56:32,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,836 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-23 19:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-23 19:56:32,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/01f002db961042fb84c5b856ac0d87b0 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0 2023-03-23 19:56:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-23 19:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,837 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-23 19:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,839 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-23 19:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-23 19:56:32,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0, entries=200, sequenceid=488, filesize=32.3 K 2023-03-23 19:56:32,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/dc0b360cad154a6aa2c53d9dbb5c8bc9 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9 2023-03-23 19:56:32,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,841 DEBUG [(jenkins-hbase4.apache.org,39351,1679601216784)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-23 19:56:32,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-23 19:56:32,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-23 19:56:32,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-23 19:56:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-23 19:56:32,841 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-23 19:56:32,842 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-23 19:56:32,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-23 19:56:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-23 19:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-23 19:56:32,842 DEBUG [Thread-3442] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-23 19:56:32,843 DEBUG [Thread-3442] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-23 19:56:32,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9, entries=150, sequenceid=488, filesize=11.9 K 2023-03-23 19:56:32,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/41cb1dc7948d418286f3520c8c950b0d as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d 2023-03-23 19:56:32,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d, entries=150, sequenceid=488, filesize=11.9 K 2023-03-23 19:56:32,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 92074693d0baf272335970fad69a35c6 in 866ms, sequenceid=488, compaction requested=true 2023-03-23 19:56:32,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:32,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:32,849 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:32,849 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:32,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:32,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:32,850 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86132 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:32,850 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/A is initiating minor compaction (all files) 2023-03-23 19:56:32,850 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:32,850 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/A in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,850 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/B is initiating minor compaction (all files) 2023-03-23 19:56:32,850 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/67a48ee9f8d04e10a6dc1020d894cb21, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=84.1 K 2023-03-23 19:56:32,850 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/B in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:32,850 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/35b5336a04264daab35170463314202a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=36.6 K 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 67a48ee9f8d04e10a6dc1020d894cb21, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390189 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 35b5336a04264daab35170463314202a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390189 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 480fd8f8575b40dc9ef981b1340480d8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679601390240 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting d12e150a3d41440c93d351f1ae0b63d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679601390240 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] compactions.Compactor(207): Compacting 01f002db961042fb84c5b856ac0d87b0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679601391365 2023-03-23 19:56:32,851 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting dc0b360cad154a6aa2c53d9dbb5c8bc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679601391365 2023-03-23 19:56:32,857 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#B#compaction#561 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:32,874 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#A#compaction#562 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:32,897 DEBUG [Thread-3445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:60301 2023-03-23 19:56:32,897 DEBUG [Thread-3445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:32,898 DEBUG [Thread-3447] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:60301 2023-03-23 19:56:32,898 DEBUG [Thread-3447] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:32,899 DEBUG [Thread-3451] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62d52321 to 127.0.0.1:60301 2023-03-23 19:56:32,899 DEBUG [Thread-3451] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:32,899 DEBUG [Thread-3449] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:60301 2023-03-23 19:56:32,899 DEBUG [Thread-3449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:32,900 DEBUG [Thread-3443] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:60301 2023-03-23 19:56:32,900 DEBUG [Thread-3443] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:32,943 DEBUG [Thread-3442] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-23 19:56:32,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-23 19:56:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34277] regionserver.HRegion(9158): Flush requested on 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:33,120 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-23 19:56:33,120 DEBUG [Thread-3434] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:60301 2023-03-23 19:56:33,120 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:33,121 DEBUG [Thread-3434] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:33,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:33,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:33,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:33,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:33,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-23 19:56:33,123 DEBUG [Thread-3438] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:60301 2023-03-23 19:56:33,123 DEBUG [Thread-3438] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:33,277 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/4a7e6c2ff1f7464fbc7f405dd4c322c1 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4a7e6c2ff1f7464fbc7f405dd4c322c1 2023-03-23 19:56:33,280 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/B of 92074693d0baf272335970fad69a35c6 into 4a7e6c2ff1f7464fbc7f405dd4c322c1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:33,280 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:33,280 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/B, priority=13, startTime=1679601392849; duration=0sec 2023-03-23 19:56:33,280 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-23 19:56:33,280 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-23 19:56:33,281 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-23 19:56:33,281 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1912): 92074693d0baf272335970fad69a35c6/C is initiating minor compaction (all files) 2023-03-23 19:56:33,281 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92074693d0baf272335970fad69a35c6/C in TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:33,281 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7568cbef1c244966beb39c0e35210133, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d] into tmpdir=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp, totalSize=36.6 K 2023-03-23 19:56:33,281 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 7568cbef1c244966beb39c0e35210133, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679601390189 2023-03-23 19:56:33,281 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting cea9185c292e4920a796b27023c819bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1679601390240 2023-03-23 19:56:33,282 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] compactions.Compactor(207): Compacting 41cb1dc7948d418286f3520c8c950b0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679601391365 2023-03-23 19:56:33,285 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] throttle.PressureAwareThroughputController(145): 92074693d0baf272335970fad69a35c6#C#compaction#564 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-23 19:56:33,293 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/8d79de618d104b0384ad184adc01306b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/8d79de618d104b0384ad184adc01306b 2023-03-23 19:56:33,296 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/A of 92074693d0baf272335970fad69a35c6 into 8d79de618d104b0384ad184adc01306b(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:33,296 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:33,296 INFO [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/A, priority=13, startTime=1679601392849; duration=0sec 2023-03-23 19:56:33,296 DEBUG [RS:0;jenkins-hbase4:34277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:33,531 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323800d017fadc141fdab01f8d2f457c358 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323800d017fadc141fdab01f8d2f457c358 2023-03-23 19:56:33,692 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/650504ea263c4cadb3ce493a47c81535 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/650504ea263c4cadb3ce493a47c81535 2023-03-23 19:56:33,695 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92074693d0baf272335970fad69a35c6/C of 92074693d0baf272335970fad69a35c6 into 650504ea263c4cadb3ce493a47c81535(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-23 19:56:33,695 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:33,695 INFO [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6., storeName=92074693d0baf272335970fad69a35c6/C, priority=13, startTime=1679601392849; duration=0sec 2023-03-23 19:56:33,695 DEBUG [RS:0;jenkins-hbase4:34277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-23 19:56:33,935 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=509, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/7f1a75d11e4945529e083979be449477 2023-03-23 19:56:34,076 DEBUG [Thread-3440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:60301 2023-03-23 19:56:34,076 DEBUG [Thread-3440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:34,090 DEBUG [Thread-3436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:60301 2023-03-23 19:56:34,090 DEBUG [Thread-3436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:34,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/ed1cedbbd485410faf77335820d8ff17 2023-03-23 19:56:34,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/751dc28342214554abb7a1495f650b1b 2023-03-23 19:56:34,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/7f1a75d11e4945529e083979be449477 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7f1a75d11e4945529e083979be449477 2023-03-23 19:56:34,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7f1a75d11e4945529e083979be449477, entries=150, sequenceid=509, filesize=25.4 K 2023-03-23 19:56:34,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/ed1cedbbd485410faf77335820d8ff17 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/ed1cedbbd485410faf77335820d8ff17 2023-03-23 19:56:34,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/ed1cedbbd485410faf77335820d8ff17, entries=150, sequenceid=509, filesize=11.9 K 2023-03-23 19:56:34,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/751dc28342214554abb7a1495f650b1b as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/751dc28342214554abb7a1495f650b1b 2023-03-23 19:56:34,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/751dc28342214554abb7a1495f650b1b, entries=150, sequenceid=509, filesize=11.9 K 2023-03-23 19:56:34,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for 92074693d0baf272335970fad69a35c6 in 1642ms, sequenceid=509, compaction requested=false 2023-03-23 19:56:34,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:35,764 DEBUG [Thread-3432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:60301 2023-03-23 19:56:35,764 DEBUG [Thread-3432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:35,764 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 128 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5192 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5052 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5082 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 5221 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(398): read 4997 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-23 19:56:35,765 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:56:35,765 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:60301 2023-03-23 19:56:35,765 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:35,766 INFO [Listener at localhost/45187] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-23 19:56:35,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-23 19:56:35,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:35,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:35,769 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601395769"}]},"ts":"1679601395769"} 2023-03-23 19:56:35,769 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-23 19:56:35,773 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-23 19:56:35,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, UNASSIGN}] 2023-03-23 19:56:35,774 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, UNASSIGN 2023-03-23 19:56:35,775 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:35,775 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601395775"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679601395775"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679601395775"}]},"ts":"1679601395775"} 2023-03-23 19:56:35,776 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015}] 2023-03-23 19:56:35,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:35,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 92074693d0baf272335970fad69a35c6, disabling compactions & flushes 2023-03-23 19:56:35,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. after waiting 0 ms 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:35,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 92074693d0baf272335970fad69a35c6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=A 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=B 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92074693d0baf272335970fad69a35c6, store=C 2023-03-23 19:56:35,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-23 19:56:36,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:36,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230323476b19f612a3455b9889a98bfa0d4ef2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323476b19f612a3455b9889a98bfa0d4ef2 2023-03-23 19:56:36,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:36,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=519, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/2220ca4667ed4ec4afbc33cbb6d063ec 2023-03-23 19:56:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:37,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/6d4e4366b2c24f0d85f3bec2fbdd5f40 2023-03-23 19:56:37,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c255ba5011ea410b8106ec855b9d352f 2023-03-23 19:56:37,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/A/2220ca4667ed4ec4afbc33cbb6d063ec as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2220ca4667ed4ec4afbc33cbb6d063ec 2023-03-23 19:56:37,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2220ca4667ed4ec4afbc33cbb6d063ec, entries=100, sequenceid=519, filesize=18.6 K 2023-03-23 19:56:37,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/B/6d4e4366b2c24f0d85f3bec2fbdd5f40 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/6d4e4366b2c24f0d85f3bec2fbdd5f40 2023-03-23 19:56:37,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/6d4e4366b2c24f0d85f3bec2fbdd5f40, entries=100, sequenceid=519, filesize=9.5 K 2023-03-23 19:56:37,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/.tmp/C/c255ba5011ea410b8106ec855b9d352f as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c255ba5011ea410b8106ec855b9d352f 2023-03-23 19:56:37,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c255ba5011ea410b8106ec855b9d352f, entries=100, sequenceid=519, filesize=9.5 K 2023-03-23 19:56:37,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 92074693d0baf272335970fad69a35c6 in 1641ms, sequenceid=519, compaction requested=true 2023-03-23 19:56:37,592 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09712bb24cb840b1af26625b62251cb2, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/11843d5ac0e142b49b9da72791026786, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/10ae53457728457085ff5d7d31d81e39, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/b1bb745e8b384d90b2bc5534a78b4050, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5d862ecb361345f2b2398705e5e728cf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/67a48ee9f8d04e10a6dc1020d894cb21, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0] to archive 2023-03-23 19:56:37,592 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:37,594 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/871d42a26daf4debbe6e1ef2b5bf42f6 2023-03-23 19:56:37,595 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5ee736e9d0254f22b0d6070b0238374b 2023-03-23 19:56:37,596 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/aa998607cea14b02b7e4ddd746f9cb4f 2023-03-23 19:56:37,597 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/494bcd2e9719488eb2e976f64989ca91 2023-03-23 19:56:37,598 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09712bb24cb840b1af26625b62251cb2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09712bb24cb840b1af26625b62251cb2 2023-03-23 19:56:37,599 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e5349669ea814243befde008445866f1 2023-03-23 19:56:37,600 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6fd393b144e5424d886c47290713feb8 2023-03-23 19:56:37,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/d378a905a1ad447eada8fa053c9e1bd6 2023-03-23 19:56:37,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/11843d5ac0e142b49b9da72791026786 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/11843d5ac0e142b49b9da72791026786 2023-03-23 19:56:37,602 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e98a6bf2a0944a2285b147f5f957b834 2023-03-23 19:56:37,603 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/10ae53457728457085ff5d7d31d81e39 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/10ae53457728457085ff5d7d31d81e39 2023-03-23 19:56:37,604 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/6cc744d5b0a241ca89128ab9996ffc7f 2023-03-23 19:56:37,605 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/a329a16f29924491885107a073eb74d1 2023-03-23 19:56:37,606 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7ff95642f8aa4aeb98f3f2f7cf35720e 2023-03-23 19:56:37,606 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/da2b9b6ec1874a80b0ed0fd18c6d3ff0 2023-03-23 19:56:37,607 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/58e5a1fcea0844dab40b9d83c2d55759 2023-03-23 19:56:37,608 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/4edba96c1e4e4b208142e0c569f346a0 2023-03-23 19:56:37,609 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/3aa7659192cf4d679fdba6f4cffafc30 2023-03-23 19:56:37,610 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/09945be1e71948099c2a92b283307afe 2023-03-23 19:56:37,611 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/b1bb745e8b384d90b2bc5534a78b4050 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/b1bb745e8b384d90b2bc5534a78b4050 2023-03-23 19:56:37,612 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2aabda79beec47ef8c6c3db291e2d1fe 2023-03-23 19:56:37,613 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/f953deabb393418f93e885bd10f0c07c 2023-03-23 19:56:37,613 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/e458d3efbecd45ec94ff49dbe2a494e1 2023-03-23 19:56:37,614 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/9a2c5fa7f7fd4a578413590319482305 2023-03-23 19:56:37,615 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/72efc34bc4954f23863ad44fa78daefe 2023-03-23 19:56:37,616 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5d862ecb361345f2b2398705e5e728cf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/5d862ecb361345f2b2398705e5e728cf 2023-03-23 19:56:37,617 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/85427ae05cca4b449c0bec49d6dc5e72 2023-03-23 19:56:37,617 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/cc414a9b867f491ab4d94f931be47f6a 2023-03-23 19:56:37,618 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/67a48ee9f8d04e10a6dc1020d894cb21 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/67a48ee9f8d04e10a6dc1020d894cb21 2023-03-23 19:56:37,619 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/480fd8f8575b40dc9ef981b1340480d8 2023-03-23 19:56:37,620 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/01f002db961042fb84c5b856ac0d87b0 2023-03-23 19:56:37,622 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/be09fc6cd0f7495489bf63a61e22abfc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1c17c185f8bd459bab6facfafde8f563, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/9e521e1ad94c406cbd002c6d28ba349b, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3555567dd5b04c13af5729d3c8a991ae, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3c8a9b6ccf49463d9eb9d62b135cc21e, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f6b5dfb5d0aa4846bf7532de1a933d1f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/35b5336a04264daab35170463314202a, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9] to archive 2023-03-23 19:56:37,623 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:37,624 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d01f6df6867e442d8552cb99ed8cbb3c 2023-03-23 19:56:37,624 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4fadf89f275f496194537176931016ad 2023-03-23 19:56:37,625 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4788f02deddd431595bb483171c2e721 2023-03-23 19:56:37,626 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/be09fc6cd0f7495489bf63a61e22abfc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/be09fc6cd0f7495489bf63a61e22abfc 2023-03-23 19:56:37,627 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/33ab07337fcf4e79b355a529019847dd 2023-03-23 19:56:37,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/5195987f6b0b4766aa9e79764f645ac4 2023-03-23 19:56:37,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/7396d94e9bab4ce1b35b540ae383fa33 2023-03-23 19:56:37,629 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1c17c185f8bd459bab6facfafde8f563 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1c17c185f8bd459bab6facfafde8f563 2023-03-23 19:56:37,630 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00b0fcce60a34cc796c1a27b7d1e9ec0 2023-03-23 19:56:37,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/00ab48aa2eb0470fb54b489333e1f4a3 2023-03-23 19:56:37,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/9e521e1ad94c406cbd002c6d28ba349b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/9e521e1ad94c406cbd002c6d28ba349b 2023-03-23 19:56:37,632 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/06055cc5d2ad476cbe42b3d0b851d8bc 2023-03-23 19:56:37,633 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4f10cf2547a64dd8a357ed580a48b070 2023-03-23 19:56:37,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3555567dd5b04c13af5729d3c8a991ae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3555567dd5b04c13af5729d3c8a991ae 2023-03-23 19:56:37,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/8a4cbd58e60348ba99921e9b6e25995f 2023-03-23 19:56:37,635 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/da941e9c67af448e8ed57412846b6f76 2023-03-23 19:56:37,636 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d070fbd03a0c4782a3a8df1945468a99 2023-03-23 19:56:37,637 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/c2e4a60aaa69453fb7924faecbedd6d1 2023-03-23 19:56:37,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/0f90c0f42d9d45619a0a6dde04353c9e 2023-03-23 19:56:37,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3c8a9b6ccf49463d9eb9d62b135cc21e to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/3c8a9b6ccf49463d9eb9d62b135cc21e 2023-03-23 19:56:37,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/31634e14e5c447bdb65901585a03eebc 2023-03-23 19:56:37,640 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/59f939a9f3ef400ca7dee4df9c754ff0 2023-03-23 19:56:37,641 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/1d06205127954d0dbd193a0a4e06ca1d 2023-03-23 19:56:37,642 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/975c52a8ce99410b99b3294b71c22dbf 2023-03-23 19:56:37,643 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f6b5dfb5d0aa4846bf7532de1a933d1f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f6b5dfb5d0aa4846bf7532de1a933d1f 2023-03-23 19:56:37,644 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/f2f7a276b2ed48638ee47b6b2594b2ed 2023-03-23 19:56:37,645 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/cf8f0116b02743b4b65c0eed5d86dcdc 2023-03-23 19:56:37,646 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/35b5336a04264daab35170463314202a to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/35b5336a04264daab35170463314202a 2023-03-23 19:56:37,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/76ac73c8e5134b29a54b3ffc307ec81c 2023-03-23 19:56:37,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/d12e150a3d41440c93d351f1ae0b63d1 2023-03-23 19:56:37,648 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/dc0b360cad154a6aa2c53d9dbb5c8bc9 2023-03-23 19:56:37,650 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7ee37bc14f74a53a62e165890e3d5bb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d18eb2c9ce574beba69c3b16d770b71c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d99a2345122a4b3583ac95bb651e0722, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/4b0f4d96f49648ebacef15aef34f9cbf, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29524add966a40649a9d7d07b3f91a60, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/27791058d4a348d3aaf539685ef00c38, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7568cbef1c244966beb39c0e35210133, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d] to archive 2023-03-23 19:56:37,651 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-23 19:56:37,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cf78b69aa334e32abdef44f230a6caf 2023-03-23 19:56:37,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/73f1910185854d3eb22c16b380b1497f 2023-03-23 19:56:37,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9d5ca36aebc84b24bd5b2278e1d817df 2023-03-23 19:56:37,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7ee37bc14f74a53a62e165890e3d5bb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7ee37bc14f74a53a62e165890e3d5bb 2023-03-23 19:56:37,655 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/904e30256a504b90afceee2a252de882 2023-03-23 19:56:37,656 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/2c15a5b3cd6b4e11900ab55c1b1df4ce 2023-03-23 19:56:37,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5291bb782fde4ae8a8814ef6c8de96fd 2023-03-23 19:56:37,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d18eb2c9ce574beba69c3b16d770b71c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d18eb2c9ce574beba69c3b16d770b71c 2023-03-23 19:56:37,658 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/a8a204dbcfc64af090d2c1a3ee41ef51 2023-03-23 19:56:37,659 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5a07aff26af04f36bf1ff782ebb66aba 2023-03-23 19:56:37,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d99a2345122a4b3583ac95bb651e0722 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/d99a2345122a4b3583ac95bb651e0722 2023-03-23 19:56:37,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29280553a3524efc92baba37fe6801a9 2023-03-23 19:56:37,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/9da37a06f4514a7487ebe9cde61b1ece 2023-03-23 19:56:37,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/4b0f4d96f49648ebacef15aef34f9cbf to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/4b0f4d96f49648ebacef15aef34f9cbf 2023-03-23 19:56:37,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/dae000bd25cf49308f313beefccba3d1 2023-03-23 19:56:37,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/5bf93a478eff47d9afa3636f5b6ad51c 2023-03-23 19:56:37,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/81953243e2c14799994ed087c8b45b64 2023-03-23 19:56:37,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3cb682ba7f6a4dd6a5c43f071b863bfd 2023-03-23 19:56:37,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/273f130cb12e47a9ba3ecf34e2f34c13 2023-03-23 19:56:37,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29524add966a40649a9d7d07b3f91a60 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/29524add966a40649a9d7d07b3f91a60 2023-03-23 19:56:37,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c6fb3a1a8bf7424384b0d6dacff7eaf9 2023-03-23 19:56:37,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/3ddd99300f3c47a183decd5e27860298 2023-03-23 19:56:37,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/31f12e0389e847339f34a623f5f3fe80 2023-03-23 19:56:37,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c7f0a39abb5a43cfb772d5eb3ea61821 2023-03-23 19:56:37,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/27791058d4a348d3aaf539685ef00c38 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/27791058d4a348d3aaf539685ef00c38 2023-03-23 19:56:37,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/562a9712e9b543ffa8631c6ad11a7842 2023-03-23 19:56:37,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7a201b52895c434ab3f0b27331a79805 2023-03-23 19:56:37,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7568cbef1c244966beb39c0e35210133 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/7568cbef1c244966beb39c0e35210133 2023-03-23 19:56:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/b08c42066d8f464c839ee45fae7679eb 2023-03-23 19:56:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/cea9185c292e4920a796b27023c819bc 2023-03-23 19:56:37,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/41cb1dc7948d418286f3520c8c950b0d 2023-03-23 19:56:37,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits/522.seqid, newMaxSeqId=522, maxSeqId=4 2023-03-23 19:56:37,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6. 2023-03-23 19:56:37,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 92074693d0baf272335970fad69a35c6: 2023-03-23 19:56:37,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 92074693d0baf272335970fad69a35c6 2023-03-23 19:56:37,683 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=92074693d0baf272335970fad69a35c6, regionState=CLOSED 2023-03-23 19:56:37,683 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679601397683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679601397683"}]},"ts":"1679601397683"} 2023-03-23 19:56:37,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-23 19:56:37,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 92074693d0baf272335970fad69a35c6, server=jenkins-hbase4.apache.org,34277,1679601218015 in 1.9090 sec 2023-03-23 19:56:37,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-23 19:56:37,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92074693d0baf272335970fad69a35c6, UNASSIGN in 1.9120 sec 2023-03-23 19:56:37,686 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679601397686"}]},"ts":"1679601397686"} 2023-03-23 19:56:37,687 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-23 19:56:37,693 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-23 19:56:37,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9270 sec 2023-03-23 19:56:37,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-23 19:56:37,872 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-23 19:56:37,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-23 19:56:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,875 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-23 19:56:37,876 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,879 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:37,881 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C, FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits] 2023-03-23 19:56:37,883 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2220ca4667ed4ec4afbc33cbb6d063ec to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/2220ca4667ed4ec4afbc33cbb6d063ec 2023-03-23 19:56:37,884 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7f1a75d11e4945529e083979be449477 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/7f1a75d11e4945529e083979be449477 2023-03-23 19:56:37,884 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/8d79de618d104b0384ad184adc01306b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/A/8d79de618d104b0384ad184adc01306b 2023-03-23 19:56:37,886 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4a7e6c2ff1f7464fbc7f405dd4c322c1 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/4a7e6c2ff1f7464fbc7f405dd4c322c1 2023-03-23 19:56:37,887 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/6d4e4366b2c24f0d85f3bec2fbdd5f40 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/6d4e4366b2c24f0d85f3bec2fbdd5f40 2023-03-23 19:56:37,888 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/ed1cedbbd485410faf77335820d8ff17 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/B/ed1cedbbd485410faf77335820d8ff17 2023-03-23 19:56:37,889 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/650504ea263c4cadb3ce493a47c81535 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/650504ea263c4cadb3ce493a47c81535 2023-03-23 19:56:37,890 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/751dc28342214554abb7a1495f650b1b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/751dc28342214554abb7a1495f650b1b 2023-03-23 19:56:37,891 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c255ba5011ea410b8106ec855b9d352f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/C/c255ba5011ea410b8106ec855b9d352f 2023-03-23 19:56:37,893 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits/522.seqid to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6/recovered.edits/522.seqid 2023-03-23 19:56:37,893 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/.tmp/data/default/TestAcidGuarantees/92074693d0baf272335970fad69a35c6 2023-03-23 19:56:37,894 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-23 19:56:37,894 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:56:37,894 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-23 19:56:37,896 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032308d372abde7e4dd4bb994a6a4bbec7d8 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032308d372abde7e4dd4bb994a6a4bbec7d8 2023-03-23 19:56:37,897 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032312a96aa0ed134fd584cb3a4feaa6f939 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032312a96aa0ed134fd584cb3a4feaa6f939 2023-03-23 19:56:37,898 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323198aea3e61084d9898612faf00710f0f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323198aea3e61084d9898612faf00710f0f 2023-03-23 19:56:37,899 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233adb463655454fe8baf95c8a768de76b to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233adb463655454fe8baf95c8a768de76b 2023-03-23 19:56:37,900 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233c05a676a6c84e8fb9dc0c5068407032 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303233c05a676a6c84e8fb9dc0c5068407032 2023-03-23 19:56:37,901 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323476b19f612a3455b9889a98bfa0d4ef2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323476b19f612a3455b9889a98bfa0d4ef2 2023-03-23 19:56:37,901 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234c48fc12bb3146e3992a27634536f97d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303234c48fc12bb3146e3992a27634536f97d 2023-03-23 19:56:37,902 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235031962b428b411fba967c2adba42bd5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235031962b428b411fba967c2adba42bd5 2023-03-23 19:56:37,903 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032355f297373cb042d899ab063f3f3adaa2 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032355f297373cb042d899ab063f3f3adaa2 2023-03-23 19:56:37,904 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323565d29009e1e4491abda6f517969f242 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323565d29009e1e4491abda6f517969f242 2023-03-23 19:56:37,904 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358b7cbbac57f410eb907cd66b3804f71 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032358b7cbbac57f410eb907cd66b3804f71 2023-03-23 19:56:37,905 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032359e93ae33721412f804988eda8bd7dcd to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032359e93ae33721412f804988eda8bd7dcd 2023-03-23 19:56:37,906 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235d184dbbbe1f44a682e07b0d041aa5c7 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303235d184dbbbe1f44a682e07b0d041aa5c7 2023-03-23 19:56:37,907 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368246333e3f941bf934ec09bff06e8a5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032368246333e3f941bf934ec09bff06e8a5 2023-03-23 19:56:37,908 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323772793a5ac0543a8abd0e1b64095932d to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323772793a5ac0543a8abd0e1b64095932d 2023-03-23 19:56:37,908 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032377e92526016441a98ce1040098b5361f to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032377e92526016441a98ce1040098b5361f 2023-03-23 19:56:37,909 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323800d017fadc141fdab01f8d2f457c358 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323800d017fadc141fdab01f8d2f457c358 2023-03-23 19:56:37,910 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032399be9bfd52a14f64a68645795d4f7eb5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032399be9bfd52a14f64a68645795d4f7eb5 2023-03-23 19:56:37,911 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239a2724b613ad4cf29c0ac7e9c38b83be to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303239a2724b613ad4cf29c0ac7e9c38b83be 2023-03-23 19:56:37,912 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a0d644c7def641b2ad53e64c8e3cca43 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323a0d644c7def641b2ad53e64c8e3cca43 2023-03-23 19:56:37,912 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b1bd8c1b8eb84beb9799c17dbf191905 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b1bd8c1b8eb84beb9799c17dbf191905 2023-03-23 19:56:37,913 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b362606dd18142bea570e9b2c38354f5 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b362606dd18142bea570e9b2c38354f5 2023-03-23 19:56:37,914 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b73b1155d8194dc7ab71042c2da6f870 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323b73b1155d8194dc7ab71042c2da6f870 2023-03-23 19:56:37,914 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323cb38b7df5f4041eeb5831c076280c374 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323cb38b7df5f4041eeb5831c076280c374 2023-03-23 19:56:37,915 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d50f7c431bfb4554a32375b7f198a5ae to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323d50f7c431bfb4554a32375b7f198a5ae 2023-03-23 19:56:37,916 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb6b14640d9744ff990a2740517c9205 to hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230323eb6b14640d9744ff990a2740517c9205 2023-03-23 19:56:37,916 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-23 19:56:37,918 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,919 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-23 19:56:37,920 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-23 19:56:37,921 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,921 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-23 19:56:37,921 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679601397921"}]},"ts":"9223372036854775807"} 2023-03-23 19:56:37,922 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-23 19:56:37,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 92074693d0baf272335970fad69a35c6, NAME => 'TestAcidGuarantees,,1679601370229.92074693d0baf272335970fad69a35c6.', STARTKEY => '', ENDKEY => ''}] 2023-03-23 19:56:37,922 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-23 19:56:37,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679601397922"}]},"ts":"9223372036854775807"} 2023-03-23 19:56:37,923 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-23 19:56:37,925 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-23 19:56:37,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-03-23 19:56:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-23 19:56:37,976 INFO [Listener at localhost/45187] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-23 19:56:37,988 INFO [Listener at localhost/45187] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 225) - Thread LEAK? -, OpenFileDescriptor=591 (was 518) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=295 (was 246) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5602 (was 5503) - AvailableMemoryMB LEAK? - 2023-03-23 19:56:37,988 INFO [Listener at localhost/45187] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-23 19:56:37,988 INFO [Listener at localhost/45187] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-23 19:56:37,989 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e324982 to 127.0.0.1:60301 2023-03-23 19:56:37,989 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:37,989 DEBUG [Listener at localhost/45187] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-23 19:56:37,989 DEBUG [Listener at localhost/45187] util.JVMClusterUtil(257): Found active master hash=1085830786, stopped=false 2023-03-23 19:56:37,989 INFO [Listener at localhost/45187] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:56:37,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-23 19:56:37,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-23 19:56:37,990 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:56:37,990 INFO [Listener at localhost/45187] procedure2.ProcedureExecutor(629): Stopping 2023-03-23 19:56:37,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-23 19:56:37,991 DEBUG [Listener at localhost/45187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x237d4a4e to 127.0.0.1:60301 2023-03-23 19:56:37,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-23 19:56:37,991 DEBUG [Listener at localhost/45187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:37,991 INFO [Listener at localhost/45187] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34277,1679601218015' ***** 2023-03-23 19:56:37,991 INFO [Listener at localhost/45187] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-23 19:56:37,992 INFO [RS:0;jenkins-hbase4:34277] regionserver.HeapMemoryManager(220): Stopping 2023-03-23 19:56:37,992 INFO [RS:0;jenkins-hbase4:34277] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-23 19:56:37,992 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-23 19:56:37,992 INFO [RS:0;jenkins-hbase4:34277] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-23 19:56:37,992 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(3304): Received CLOSE for 72272839a6ffef279ba4e656fef71504 2023-03-23 19:56:37,993 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:37,993 DEBUG [RS:0;jenkins-hbase4:34277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c47425d to 127.0.0.1:60301 2023-03-23 19:56:37,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 72272839a6ffef279ba4e656fef71504, disabling compactions & flushes 2023-03-23 19:56:37,993 DEBUG [RS:0;jenkins-hbase4:34277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:37,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:56:37,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:56:37,994 INFO [RS:0;jenkins-hbase4:34277] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-23 19:56:37,994 INFO [RS:0;jenkins-hbase4:34277] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-23 19:56:37,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. after waiting 0 ms 2023-03-23 19:56:37,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:56:37,994 INFO [RS:0;jenkins-hbase4:34277] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-23 19:56:37,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 72272839a6ffef279ba4e656fef71504 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-23 19:56:37,994 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-23 19:56:37,994 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-23 19:56:37,995 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-23 19:56:37,995 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 72272839a6ffef279ba4e656fef71504=hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504.} 2023-03-23 19:56:37,995 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-23 19:56:37,995 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-23 19:56:37,995 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-23 19:56:37,995 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-23 19:56:37,995 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-23 19:56:37,997 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740, 72272839a6ffef279ba4e656fef71504 2023-03-23 19:56:38,067 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-23 19:56:38,197 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740, 72272839a6ffef279ba4e656fef71504 2023-03-23 19:56:38,295 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-23 19:56:38,295 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-23 19:56:38,397 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740, 72272839a6ffef279ba4e656fef71504 2023-03-23 19:56:38,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/.tmp/info/7f7b7686bb5c450da2774858f8db383a 2023-03-23 19:56:38,402 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/info/ab634852e7484d00bf05731a5939c5fc 2023-03-23 19:56:38,407 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ab634852e7484d00bf05731a5939c5fc 2023-03-23 19:56:38,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/.tmp/info/7f7b7686bb5c450da2774858f8db383a as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/info/7f7b7686bb5c450da2774858f8db383a 2023-03-23 19:56:38,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/info/7f7b7686bb5c450da2774858f8db383a, entries=2, sequenceid=6, filesize=4.8 K 2023-03-23 19:56:38,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 72272839a6ffef279ba4e656fef71504 in 417ms, sequenceid=6, compaction requested=false 2023-03-23 19:56:38,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/namespace/72272839a6ffef279ba4e656fef71504/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-23 19:56:38,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:56:38,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 72272839a6ffef279ba4e656fef71504: 2023-03-23 19:56:38,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679601220687.72272839a6ffef279ba4e656fef71504. 2023-03-23 19:56:38,598 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-23 19:56:38,798 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-23 19:56:38,820 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/rep_barrier/d4fd0f3ba39b405d92d14dae020b8cf6 2023-03-23 19:56:38,823 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d4fd0f3ba39b405d92d14dae020b8cf6 2023-03-23 19:56:38,998 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-23 19:56:38,998 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-23 19:56:38,998 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-23 19:56:39,198 DEBUG [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-23 19:56:39,229 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/table/d6abfce3c2404055a86d7b0c271b02c3 2023-03-23 19:56:39,231 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d6abfce3c2404055a86d7b0c271b02c3 2023-03-23 19:56:39,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/info/ab634852e7484d00bf05731a5939c5fc as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/info/ab634852e7484d00bf05731a5939c5fc 2023-03-23 19:56:39,234 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ab634852e7484d00bf05731a5939c5fc 2023-03-23 19:56:39,234 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/info/ab634852e7484d00bf05731a5939c5fc, entries=22, sequenceid=93, filesize=7.1 K 2023-03-23 19:56:39,236 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/rep_barrier/d4fd0f3ba39b405d92d14dae020b8cf6 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/rep_barrier/d4fd0f3ba39b405d92d14dae020b8cf6 2023-03-23 19:56:39,238 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d4fd0f3ba39b405d92d14dae020b8cf6 2023-03-23 19:56:39,238 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/rep_barrier/d4fd0f3ba39b405d92d14dae020b8cf6, entries=6, sequenceid=93, filesize=5.5 K 2023-03-23 19:56:39,239 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/.tmp/table/d6abfce3c2404055a86d7b0c271b02c3 as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/table/d6abfce3c2404055a86d7b0c271b02c3 2023-03-23 19:56:39,241 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d6abfce3c2404055a86d7b0c271b02c3 2023-03-23 19:56:39,241 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/table/d6abfce3c2404055a86d7b0c271b02c3, entries=9, sequenceid=93, filesize=5.5 K 2023-03-23 19:56:39,242 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1247ms, sequenceid=93, compaction requested=false 2023-03-23 19:56:39,248 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-23 19:56:39,249 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-23 19:56:39,250 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-23 19:56:39,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-23 19:56:39,250 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-23 19:56:39,372 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-23 19:56:39,399 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34277,1679601218015; all regions closed. 2023-03-23 19:56:39,405 DEBUG [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/oldWALs 2023-03-23 19:56:39,405 INFO [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34277%2C1679601218015.meta:.meta(num 1679601220423) 2023-03-23 19:56:39,409 DEBUG [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/oldWALs 2023-03-23 19:56:39,409 INFO [RS:0;jenkins-hbase4:34277] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34277%2C1679601218015:(num 1679601219445) 2023-03-23 19:56:39,409 DEBUG [RS:0;jenkins-hbase4:34277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:39,409 INFO [RS:0;jenkins-hbase4:34277] regionserver.LeaseManager(133): Closed leases 2023-03-23 19:56:39,409 INFO [RS:0;jenkins-hbase4:34277] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-23 19:56:39,410 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-23 19:56:39,411 INFO [RS:0;jenkins-hbase4:34277] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34277 2023-03-23 19:56:39,418 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-23 19:56:39,418 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34277,1679601218015 2023-03-23 19:56:39,418 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6adf82dc rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2304] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,419 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-23 19:56:39,419 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34277,1679601218015] 2023-03-23 19:56:39,419 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@cdc0cbc rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2304] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,419 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34277,1679601218015; numProcessing=1 2023-03-23 19:56:39,423 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34277,1679601218015 already deleted, retry=false 2023-03-23 19:56:39,423 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34277,1679601218015 expired; onlineServers=0 2023-03-23 19:56:39,423 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39351,1679601216784' ***** 2023-03-23 19:56:39,423 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-23 19:56:39,424 DEBUG [M:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1857cb9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-23 19:56:39,424 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39351,1679601216784 2023-03-23 19:56:39,424 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39351,1679601216784; all regions closed. 2023-03-23 19:56:39,424 DEBUG [M:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-23 19:56:39,424 DEBUG [M:0;jenkins-hbase4:39351] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-23 19:56:39,424 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-23 19:56:39,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679601219150] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679601219150,5,FailOnTimeoutGroup] 2023-03-23 19:56:39,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679601219149] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679601219149,5,FailOnTimeoutGroup] 2023-03-23 19:56:39,424 DEBUG [M:0;jenkins-hbase4:39351] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-23 19:56:39,424 INFO [M:0;jenkins-hbase4:39351] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-23 19:56:39,424 INFO [M:0;jenkins-hbase4:39351] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-23 19:56:39,424 INFO [M:0;jenkins-hbase4:39351] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-23 19:56:39,425 DEBUG [M:0;jenkins-hbase4:39351] master.HMaster(1512): Stopping service threads 2023-03-23 19:56:39,425 INFO [M:0;jenkins-hbase4:39351] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-23 19:56:39,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-23 19:56:39,425 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-23 19:56:39,425 ERROR [M:0;jenkins-hbase4:39351] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-23 19:56:39,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-23 19:56:39,426 INFO [M:0;jenkins-hbase4:39351] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-23 19:56:39,426 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-23 19:56:39,426 DEBUG [M:0;jenkins-hbase4:39351] zookeeper.ZKUtil(398): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-23 19:56:39,426 WARN [M:0;jenkins-hbase4:39351] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-23 19:56:39,426 INFO [M:0;jenkins-hbase4:39351] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-23 19:56:39,426 INFO [M:0;jenkins-hbase4:39351] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-23 19:56:39,427 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-23 19:56:39,427 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:56:39,427 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:56:39,427 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-23 19:56:39,427 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:56:39,427 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.80 KB heapSize=393 KB 2023-03-23 19:56:39,434 INFO [M:0;jenkins-hbase4:39351] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.80 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2c092d81daf6447db79ccbf50fe5b11c 2023-03-23 19:56:39,438 INFO [M:0;jenkins-hbase4:39351] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2c092d81daf6447db79ccbf50fe5b11c 2023-03-23 19:56:39,438 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2c092d81daf6447db79ccbf50fe5b11c as hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2c092d81daf6447db79ccbf50fe5b11c 2023-03-23 19:56:39,441 INFO [M:0;jenkins-hbase4:39351] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2c092d81daf6447db79ccbf50fe5b11c 2023-03-23 19:56:39,441 INFO [M:0;jenkins-hbase4:39351] regionserver.HStore(1080): Added hdfs://localhost:46107/user/jenkins/test-data/7176bbd9-6db3-919a-fda2-a254a52fff78/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2c092d81daf6447db79ccbf50fe5b11c, entries=65, sequenceid=571, filesize=24.7 K 2023-03-23 19:56:39,442 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegion(2948): Finished flush of dataSize ~344.80 KB/353075, heapSize ~392.98 KB/402416, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 15ms, sequenceid=571, compaction requested=false 2023-03-23 19:56:39,444 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-23 19:56:39,444 DEBUG [M:0;jenkins-hbase4:39351] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-23 19:56:39,446 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-23 19:56:39,446 INFO [M:0;jenkins-hbase4:39351] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-23 19:56:39,447 INFO [M:0;jenkins-hbase4:39351] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39351 2023-03-23 19:56:39,448 DEBUG [M:0;jenkins-hbase4:39351] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39351,1679601216784 already deleted, retry=false 2023-03-23 19:56:39,522 INFO [RS:0;jenkins-hbase4:34277] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34277,1679601218015; zookeeper connection closed. 2023-03-23 19:56:39,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-23 19:56:39,522 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@74d77c65 rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2304] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,522 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5b7215f9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5b7215f9 2023-03-23 19:56:39,522 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): regionserver:34277-0x10112f0937e0001, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-23 19:56:39,522 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@27982b81 rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2304] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,523 INFO [Listener at localhost/45187] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-23 19:56:39,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-23 19:56:39,622 INFO [M:0;jenkins-hbase4:39351] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39351,1679601216784; zookeeper connection closed. 2023-03-23 19:56:39,622 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43dd588f rejected from java.util.concurrent.ThreadPoolExecutor@4d362d29[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3857] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,622 DEBUG [Listener at localhost/45187-EventThread] zookeeper.ZKWatcher(600): master:39351-0x10112f0937e0000, quorum=127.0.0.1:60301, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-23 19:56:39,622 ERROR [Listener at localhost/45187-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3218437 rejected from java.util.concurrent.ThreadPoolExecutor@4d362d29[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3857] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-23 19:56:39,624 WARN [Listener at localhost/45187] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-23 19:56:39,627 INFO [Listener at localhost/45187] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-23 19:56:39,735 WARN [BP-1032745266-172.31.14.131-1679601213371 heartbeating to localhost/127.0.0.1:46107] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-23 19:56:39,735 WARN [BP-1032745266-172.31.14.131-1679601213371 heartbeating to localhost/127.0.0.1:46107] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1032745266-172.31.14.131-1679601213371 (Datanode Uuid 8951c20b-99ff-4f66-9662-9fe0fcff31a4) service to localhost/127.0.0.1:46107 2023-03-23 19:56:39,738 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6/dfs/data/data1/current/BP-1032745266-172.31.14.131-1679601213371] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-23 19:56:39,738 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/97fa59a6-10bf-ac8e-d43a-338edcaa6101/cluster_e04f9dea-048a-0c4a-2dcc-8503cd245aa6/dfs/data/data2/current/BP-1032745266-172.31.14.131-1679601213371] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-23 19:56:39,776 INFO [Listener at localhost/45187] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-23 19:56:39,936 INFO [Listener at localhost/45187] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-23 19:56:40,087 INFO [Listener at localhost/45187] hbase.HBaseTestingUtility(1293): Minicluster is down